Supervised Learning - Week 2 / Part 1
In this part we will learn how to make linear regression algorithm much faster and powerful.
- Suppose we are interested in linear regression and we are looking for not only one feature but multiple features.
- In the previous part we saw that our dataset had one feature; x, size of the house, and we were predicting y; price of the house by using the linear regression formula:
. - What if we not only know the size of the house (
) but also the number of bedrooms (
), number of floors (
) and age of the house in years (
)?
hSize = [2104; 1416; 1534; 852];
hNumBedrooms = [5; 3; 3; 2];
hNumFloors = [1; 2; 2; 1];
hPrice = [460; 232; 315; 178];
table(hSize,hNumBedrooms,hNumFloors,hAge,hPrice)
ans = 4×5 table
| | hSize | hNumBedrooms | hNumFloors | hAge | hPrice |
|---|
| 1 | 2104 | 5 | 1 | 45 | 460 |
|---|
| 2 | 1416 | 3 | 2 | 40 | 232 |
|---|
| 3 | 1534 | 3 | 2 | 30 | 315 |
|---|
| 4 | 852 | 2 | 1 | 36 | 178 |
|---|
- Note that we represent each feature (
column in the table) with subscript j, where
feature, for
. - We represent the number of features by using n, where
for the table above. - Moreover,
represents the features of
training example (
row in the table), where
. The parameter m represents the number of examples. - Consequently, we can represent the value of
feature (column) in
training example (row) by using the notation
.
Now, we have multiple features, let's look at our how our model look like
- Previously:

- If we generalize the one feature case:

where this expression is called multi parameter linear regression. More importantly, here, both
and b are the parameters. Example: 
- Base price of a house starts at $80 000 assuming it has no size, no bedrooms, no floors and no age.
- For every addtiional squared foot, the price increases $100.
- For every addition of bedrooms, the price increases $4000.
- For every addition of floors, the price increases $10 000.
- For every additional year in houses age, the price reduces $2000.
Vectorization
We can define a case where 
b is a number
We can calculate the linear regression expression without vectorization as follows:
If the number of features is not 3 but 100 000, this would be inefficient to code as well as inefficient for the computation. An alternative approach without also vectorization becomes,
With vectorization,
or
Let's evaluate the benefits of vectorization
% Non-vectoral implementation
disp(['Non-vectoral execution time: ', num2str(tExec),' seconds.'])
Non-vectoral execution time: 0.17638 seconds.
% Vectoral implementation
disp(['Vectoral implementation execution time: ', num2str(tExec),' seconds.'])
Vectoral implementation execution time: 0.096702 seconds.
clear w x % release memory
Linear Regression with Multiple Variables
------------------------------
Single Varible (Scalar) Notation
- Inputs:

- Labels:

- Parameters: w and b
- Model:

- Cost Function:

- Gradient Descent:
and 
------------------------------
Multiple Variables (Vector) Notation
- Inputs:
represents the
row of
, where
is the
input data matrix. - Labels:

- Parameters:
and b - Model:
, for
or 
- Cost Function:
or 
- Gradient Descent:
, for
and 
------------------------------
How can we minimize the cost function in multiple variable case?
- We can calculate the gradient of the cost function w.r.t.
as follows:
, where the parial derivative
for
could be calculated as follows:
1) 
2) By using the chain and sum rules,
3) We can see from the above expression that
for
.4) Therefore,
Similarly, the partial derivative of the cost function w.r.t. the bias becomes,
5) 
6) If we solve the above expression using (2)
7) As we know that
. Hence, ------------------------------
Consequently, the policy for updating the parameters in BGD becomes:
for
and
. An Alternative Way to Gradient Descent
This method is called "Normal Equation".
Advantages
- It solves the linear regression problem for the parameters w and b without iterations.
Disadvantages
- Only works for linear regression and can't be generalized to other learning algortihms logistic regression, neural networks etc.
- Slow when number of features is large ( > 10, 000 features)
What you need to know
- Normal equation method may be used in machine learning libraries that implement linear regression.
- Gradient descent is the recommended method for finding parameters w and b.
Implentation of the Gradient Descent with Multiple Variables
Similar to the gradient descent implementation with single feature we will have two steps:
- Step 1: Initialize the
and b. - Step 2: Repeat (1) and (2) simultaneously until the convergence (or the stopping criteria is met).
We can also represent the above steps mathematically as follows:
1) Randomly initialize
and b. 2) Repeat until the stopping criteria is met:
where
and
. Note that
and b must be updated simultaneously. Note that
represents
vector, the
row of
, where
is the
input data matrix.
is the
weights vector.
Problem Statement
We will use the example of housing price prediction. The training dataset contains three examples with four features (size, bedrooms, floors and age). Please note that the size is given in sqft rather than 1000 sqft, which will cause issues and we will solve this in the following section!
Let's present the data in a table format
houseSize = [2104; 1416; 852];
housePrice = [460; 232; 178];
table(houseSize,numBedrooms,numFloors,houseAge,housePrice)
ans = 3×5 table
| | houseSize | numBedrooms | numFloors | houseAge | housePrice |
|---|
| 1 | 2104 | 5 | 1 | 45 | 460 |
|---|
| 2 | 1416 | 3 | 2 | 40 | 232 |
|---|
| 3 | 852 | 2 | 1 | 35 | 178 |
|---|
Objective: You will build a linear regression algorithm so that you can predict the price for other houses. For instance, a house with 1200 sqft, 3 bedrooms, 1 floor and 40 years old.
Let's convert the information contained in the table into a matrix. Note that the parameter "price" is what we are trying to estimate/predict. Therefore,
- X_train: Training example matrix (
), where the matrix contains n features and m examples. - y_train: Training example targets (
)
X_train = [houseSize numBedrooms numFloors houseAge]
2104 5 1 45
1416 3 2 40
852 2 1 35
Since our goal is to find the linear relationship (
and b), the linear regression model becomes,
for
, where
and
are the
column vector that consists of the linear weights and
row vector of the training matrix (X_train), respectively. The training matrix
is also given as follows.Let's implement the batch gradient descent (BGD) algorithm for multiple variables.
Notation
General Notation | Description/Style | Representaion in MATLAB |
a | scalar - non bold |
|
| vector - bold |
|
| matrix - bold capital |
|
Regression Specific |
|
|
m | number of examples/datapoints | m |
n | number of features | n |
| training example matrix | X_train |
| training example targets | y_train |
| row vector of , which correspondsto a data point with j features, where  | X_train(i,:) |
| column vector of , which corresponds to all data points for a givenfeature | X_train(:,j) |
| element of the targets vector | y_train(i) |
| parameter - weights vector | w |
b | parameter - bias | b |
| Model evaluation at the point parameterized by and b. | X_train(i,:)*w+b |
| Gradient/partial derivative of the cost function w.r.t the weight | dJdw |
| Gradient/partial derivative of the cost function w.r.t the bias b | dJdb |
The steps for BGD for multiple varibles are given by,
- Initialize the parameters; n-by-1
vector and scalar b (zero initialization is generally simple and efficient enough at this stage). - Update both
and b simultaneously. - Repeat (2) until the BGD algorithm until either maximum number of iterations is reached or the algorithm yields a cost value, which is smaller than a user selected threshold and doesn't change anymore (algorithm converges).
alpha = 5.0e-7; % Learning rate
condStop = 1e-3; % Stopping condition
maxIter = 1000; % Maximum number of iterations
% Run BGD algoritm and print the results
[w,b,J_hist] = hBatchGradientDescentMV(X_train,y_train,alpha,condStop,maxIter,verbose,verboseFreq);
Iteration #0 J = 2529.46 w_1 = 2.41e-01 w_2 = 5.59e-04 w_3 = 1.84e-04 w_4 = 6.03e-03 b = 1.45e-04
Iteration #100 J = 695.99 w_1 = 2.02e-01 w_2 = 7.98e-04 w_3 = -9.97e-04 w_4 = -2.20e-03 b = -1.20e-04
Iteration #200 J = 694.92 w_1 = 2.03e-01 w_2 = 1.13e-03 w_3 = -2.14e-03 w_4 = -9.41e-03 b = -3.60e-04
Iteration #300 J = 693.86 w_1 = 2.03e-01 w_2 = 1.46e-03 w_3 = -3.29e-03 w_4 = -1.66e-02 b = -5.98e-04
Iteration #400 J = 692.81 w_1 = 2.03e-01 w_2 = 1.78e-03 w_3 = -4.43e-03 w_4 = -2.37e-02 b = -8.36e-04
Iteration #500 J = 691.77 w_1 = 2.03e-01 w_2 = 2.11e-03 w_3 = -5.57e-03 w_4 = -3.08e-02 b = -1.07e-03
Iteration #600 J = 690.73 w_1 = 2.03e-01 w_2 = 2.44e-03 w_3 = -6.71e-03 w_4 = -3.79e-02 b = -1.31e-03
Iteration #700 J = 689.71 w_1 = 2.03e-01 w_2 = 2.77e-03 w_3 = -7.85e-03 w_4 = -4.50e-02 b = -1.54e-03
Iteration #800 J = 688.70 w_1 = 2.04e-01 w_2 = 3.10e-03 w_3 = -8.99e-03 w_4 = -5.20e-02 b = -1.77e-03
Iteration #900 J = 687.69 w_1 = 2.04e-01 w_2 = 3.43e-03 w_3 = -1.01e-02 w_4 = -5.90e-02 b = -2.01e-03
==================================
BGD Stopped: max number of iterations (999)
Let's plot the cost function against the iteration number
plot(1:length(J_hist),J_hist);
ylabel('Cost function, J(w,b)');
disp(['The parameters found by gradient descent: w = [',num2str(w(:,end).'),'], b = ',num2str(b(end))])
The parameters found by gradient descent: w = [0.20397 0.0037492 -0.011249 -0.065861], b = -0.0022354
fprintf('Example #%d - Estimated value: %.2f | Target value: %.2f\n',i, w.'*X_train(i,:).'+b, y_train(i))
end
Example #1 - Estimated value: 508.04 | Target value: 409.62
Example #4.286872e+02 - Estimated value: 424.99 | Target value: 425.71
Example #4.255719e+02 - Estimated value: 425.60 | Target value: 425.59
Example #4.255964e+02 - Estimated value: 425.60 | Target value: 425.60
Example #4.255980e+02 - Estimated value: 425.60 | Target value: 425.60
Example #4.255998e+02 - Estimated value: 425.60 | Target value: 425.60
Example #4.256017e+02 - Estimated value: 425.60 | Target value: 425.60
Example #4.256035e+02 - Estimated value: 425.60 | Target value: 425.60
Example #4.256053e+02 - Estimated value: 425.61 | Target value: 425.61
Example #4.256071e+02 - Estimated value: 425.61 | Target value: 425.61
Example #4.256090e+02 - Estimated value: 425.61 | Target value: 425.61
Example #4.256108e+02 - Estimated value: 425.61 | Target value: 425.61
Example #4.256126e+02 - Estimated value: 425.61 | Target value: 425.61
Example #4.256144e+02 - Estimated value: 425.62 | Target value: 425.62
Example #4.256162e+02 - Estimated value: 425.62 | Target value: 425.62
Example #4.256181e+02 - Estimated value: 425.62 | Target value: 425.62
Example #4.256199e+02 - Estimated value: 425.62 | Target value: 425.62
Example #4.256217e+02 - Estimated value: 425.62 | Target value: 425.62
Example #4.256235e+02 - Estimated value: 425.62 | Target value: 425.62
Example #4.256253e+02 - Estimated value: 425.63 | Target value: 425.63
Example #4.256271e+02 - Estimated value: 425.63 | Target value: 425.63
Example #4.256290e+02 - Estimated value: 425.63 | Target value: 425.63
Example #4.256308e+02 - Estimated value: 425.63 | Target value: 425.63
Example #4.256326e+02 - Estimated value: 425.63 | Target value: 425.63
Example #4.256344e+02 - Estimated value: 425.64 | Target value: 425.64
Example #4.256362e+02 - Estimated value: 425.64 | Target value: 425.64
Example #4.256381e+02 - Estimated value: 425.64 | Target value: 425.64
Example #4.256399e+02 - Estimated value: 425.64 | Target value: 425.64
Example #4.256417e+02 - Estimated value: 425.64 | Target value: 425.64
Example #4.256435e+02 - Estimated value: 425.64 | Target value: 425.64
Example #4.256453e+02 - Estimated value: 425.65 | Target value: 425.65
Example #4.256471e+02 - Estimated value: 425.65 | Target value: 425.65
Example #4.256490e+02 - Estimated value: 425.65 | Target value: 425.65
Example #4.256508e+02 - Estimated value: 425.65 | Target value: 425.65
Example #4.256526e+02 - Estimated value: 425.65 | Target value: 425.65
Example #4.256544e+02 - Estimated value: 425.66 | Target value: 425.66
Example #4.256562e+02 - Estimated value: 425.66 | Target value: 425.66
Example #4.256580e+02 - Estimated value: 425.66 | Target value: 425.66
Example #4.256598e+02 - Estimated value: 425.66 | Target value: 425.66
Example #4.256617e+02 - Estimated value: 425.66 | Target value: 425.66
Example #4.256635e+02 - Estimated value: 425.66 | Target value: 425.66
Example #4.256653e+02 - Estimated value: 425.67 | Target value: 425.67
Example #4.256671e+02 - Estimated value: 425.67 | Target value: 425.67
Example #4.256689e+02 - Estimated value: 425.67 | Target value: 425.67
Example #4.256707e+02 - Estimated value: 425.67 | Target value: 425.67
Example #4.256725e+02 - Estimated value: 425.67 | Target value: 425.67
Example #4.256744e+02 - Estimated value: 425.67 | Target value: 425.68
Example #4.256762e+02 - Estimated value: 425.68 | Target value: 425.68
Example #4.256780e+02 - Estimated value: 425.68 | Target value: 425.68
Example #4.256798e+02 - Estimated value: 425.68 | Target value: 425.68
Example #4.256816e+02 - Estimated value: 425.68 | Target value: 425.68
Example #4.256834e+02 - Estimated value: 425.68 | Target value: 425.68
Example #4.256852e+02 - Estimated value: 425.69 | Target value: 425.69
Example #4.256870e+02 - Estimated value: 425.69 | Target value: 425.69
Example #4.256888e+02 - Estimated value: 425.69 | Target value: 425.69
Example #4.256907e+02 - Estimated value: 425.69 | Target value: 425.69
Example #4.256925e+02 - Estimated value: 425.69 | Target value: 425.69
Example #4.256943e+02 - Estimated value: 425.69 | Target value: 425.70
Example #4.256961e+02 - Estimated value: 425.70 | Target value: 425.70
Example #4.256979e+02 - Estimated value: 425.70 | Target value: 425.70
Example #4.256997e+02 - Estimated value: 425.70 | Target value: 425.70
Example #4.257015e+02 - Estimated value: 425.70 | Target value: 425.70
Example #4.257033e+02 - Estimated value: 425.70 | Target value: 425.70
Example #4.257051e+02 - Estimated value: 425.71 | Target value: 425.71
Example #4.257069e+02 - Estimated value: 425.71 | Target value: 425.71
Example #4.257087e+02 - Estimated value: 425.71 | Target value: 425.71
Example #4.257106e+02 - Estimated value: 425.71 | Target value: 425.71
Example #4.257124e+02 - Estimated value: 425.71 | Target value: 425.71
Example #4.257142e+02 - Estimated value: 425.71 | Target value: 425.72
Example #4.257160e+02 - Estimated value: 425.72 | Target value: 425.72
Example #4.257178e+02 - Estimated value: 425.72 | Target value: 425.72
Example #4.257196e+02 - Estimated value: 425.72 | Target value: 425.72
Example #4.257214e+02 - Estimated value: 425.72 | Target value: 425.72
Example #4.257232e+02 - Estimated value: 425.72 | Target value: 425.72
Example #4.257250e+02 - Estimated value: 425.73 | Target value: 425.73
Example #4.257268e+02 - Estimated value: 425.73 | Target value: 425.73
Example #4.257286e+02 - Estimated value: 425.73 | Target value: 425.73
Example #4.257304e+02 - Estimated value: 425.73 | Target value: 425.73
Example #4.257322e+02 - Estimated value: 425.73 | Target value: 425.73
Example #4.257340e+02 - Estimated value: 425.73 | Target value: 425.74
Example #4.257358e+02 - Estimated value: 425.74 | Target value: 425.74
Example #4.257376e+02 - Estimated value: 425.74 | Target value: 425.74
Example #4.257394e+02 - Estimated value: 425.74 | Target value: 425.74
Example #4.257412e+02 - Estimated value: 425.74 | Target value: 425.74
Example #4.257431e+02 - Estimated value: 425.74 | Target value: 425.74
Example #4.257449e+02 - Estimated value: 425.75 | Target value: 425.75
Example #4.257467e+02 - Estimated value: 425.75 | Target value: 425.75
Example #4.257485e+02 - Estimated value: 425.75 | Target value: 425.75
Example #4.257503e+02 - Estimated value: 425.75 | Target value: 425.75
Example #4.257521e+02 - Estimated value: 425.75 | Target value: 425.75
Example #4.257539e+02 - Estimated value: 425.75 | Target value: 425.76
Example #4.257557e+02 - Estimated value: 425.76 | Target value: 425.76
Example #4.257575e+02 - Estimated value: 425.76 | Target value: 425.76
Example #4.257593e+02 - Estimated value: 425.76 | Target value: 425.76
Example #4.257611e+02 - Estimated value: 425.76 | Target value: 425.76
Example #4.257629e+02 - Estimated value: 425.76 | Target value: 425.76
Example #4.257647e+02 - Estimated value: 425.77 | Target value: 425.77
Example #4.257665e+02 - Estimated value: 425.77 | Target value: 425.77
Example #4.257683e+02 - Estimated value: 425.77 | Target value: 425.77
Example #4.257701e+02 - Estimated value: 425.77 | Target value: 425.77
Example #4.257719e+02 - Estimated value: 425.77 | Target value: 425.77
Example #4.257737e+02 - Estimated value: 425.77 | Target value: 425.77
Example #4.257755e+02 - Estimated value: 425.78 | Target value: 425.78
Example #4.257773e+02 - Estimated value: 425.78 | Target value: 425.78
Example #4.257791e+02 - Estimated value: 425.78 | Target value: 425.78
Example #4.257809e+02 - Estimated value: 425.78 | Target value: 425.78
Example #4.257827e+02 - Estimated value: 425.78 | Target value: 425.78
Example #4.257845e+02 - Estimated value: 425.79 | Target value: 425.79
Example #4.257863e+02 - Estimated value: 425.79 | Target value: 425.79
Example #4.257881e+02 - Estimated value: 425.79 | Target value: 425.79
Example #4.257899e+02 - Estimated value: 425.79 | Target value: 425.79
Example #4.257917e+02 - Estimated value: 425.79 | Target value: 425.79
Example #4.257935e+02 - Estimated value: 425.79 | Target value: 425.79
Example #4.257952e+02 - Estimated value: 425.80 | Target value: 425.80
Example #4.257970e+02 - Estimated value: 425.80 | Target value: 425.80
Example #4.257988e+02 - Estimated value: 425.80 | Target value: 425.80
Example #4.258006e+02 - Estimated value: 425.80 | Target value: 425.80
Example #4.258024e+02 - Estimated value: 425.80 | Target value: 425.80
Example #4.258042e+02 - Estimated value: 425.80 | Target value: 425.81
Example #4.258060e+02 - Estimated value: 425.81 | Target value: 425.81
Example #4.258078e+02 - Estimated value: 425.81 | Target value: 425.81
Example #4.258096e+02 - Estimated value: 425.81 | Target value: 425.81
Example #4.258114e+02 - Estimated value: 425.81 | Target value: 425.81
Example #4.258132e+02 - Estimated value: 425.81 | Target value: 425.81
Example #4.258150e+02 - Estimated value: 425.82 | Target value: 425.82
Example #4.258168e+02 - Estimated value: 425.82 | Target value: 425.82
Example #4.258186e+02 - Estimated value: 425.82 | Target value: 425.82
Example #4.258204e+02 - Estimated value: 425.82 | Target value: 425.82
Example #4.258222e+02 - Estimated value: 425.82 | Target value: 425.82
Example #4.258240e+02 - Estimated value: 425.82 | Target value: 425.83
Example #4.258257e+02 - Estimated value: 425.83 | Target value: 425.83
Example #4.258275e+02 - Estimated value: 425.83 | Target value: 425.83
Example #4.258293e+02 - Estimated value: 425.83 | Target value: 425.83
Example #4.258311e+02 - Estimated value: 425.83 | Target value: 425.83
Example #4.258329e+02 - Estimated value: 425.83 | Target value: 425.83
Example #4.258347e+02 - Estimated value: 425.84 | Target value: 425.84
Example #4.258365e+02 - Estimated value: 425.84 | Target value: 425.84
Example #4.258383e+02 - Estimated value: 425.84 | Target value: 425.84
Example #4.258401e+02 - Estimated value: 425.84 | Target value: 425.84
Example #4.258419e+02 - Estimated value: 425.84 | Target value: 425.84
Example #4.258437e+02 - Estimated value: 425.84 | Target value: 425.84
Example #4.258455e+02 - Estimated value: 425.85 | Target value: 425.85
Example #4.258472e+02 - Estimated value: 425.85 | Target value: 425.85
Example #4.258490e+02 - Estimated value: 425.85 | Target value: 425.85
Example #4.258508e+02 - Estimated value: 425.85 | Target value: 425.85
Example #4.258526e+02 - Estimated value: 425.85 | Target value: 425.85
Example #4.258544e+02 - Estimated value: 425.85 | Target value: 425.86
Example #4.258562e+02 - Estimated value: 425.86 | Target value: 425.86
Example #4.258580e+02 - Estimated value: 425.86 | Target value: 425.86
Example #4.258598e+02 - Estimated value: 425.86 | Target value: 425.86
Example #4.258615e+02 - Estimated value: 425.86 | Target value: 425.86
Example #4.258633e+02 - Estimated value: 425.86 | Target value: 425.86
Example #4.258651e+02 - Estimated value: 425.87 | Target value: 425.87
Example #4.258669e+02 - Estimated value: 425.87 | Target value: 425.87
Example #4.258687e+02 - Estimated value: 425.87 | Target value: 425.87
Example #4.258705e+02 - Estimated value: 425.87 | Target value: 425.87
Example #4.258723e+02 - Estimated value: 425.87 | Target value: 425.87
Example #4.258741e+02 - Estimated value: 425.87 | Target value: 425.88
Example #4.258758e+02 - Estimated value: 425.88 | Target value: 425.88
Example #4.258776e+02 - Estimated value: 425.88 | Target value: 425.88
Example #4.258794e+02 - Estimated value: 425.88 | Target value: 425.88
Example #4.258812e+02 - Estimated value: 425.88 | Target value: 425.88
Example #4.258830e+02 - Estimated value: 425.88 | Target value: 425.88
Example #4.258848e+02 - Estimated value: 425.89 | Target value: 425.89
Example #4.258866e+02 - Estimated value: 425.89 | Target value: 425.89
Example #4.258883e+02 - Estimated value: 425.89 | Target value: 425.89
Example #4.258901e+02 - Estimated value: 425.89 | Target value: 425.89
Example #4.258919e+02 - Estimated value: 425.89 | Target value: 425.89
Example #4.258937e+02 - Estimated value: 425.89 | Target value: 425.89
Example #4.258955e+02 - Estimated value: 425.90 | Target value: 425.90
Example #4.258973e+02 - Estimated value: 425.90 | Target value: 425.90
Example #4.258990e+02 - Estimated value: 425.90 | Target value: 425.90
Example #4.259008e+02 - Estimated value: 425.90 | Target value: 425.90
Example #4.259026e+02 - Estimated value: 425.90 | Target value: 425.90
Example #4.259044e+02 - Estimated value: 425.90 | Target value: 425.91
Example #4.259062e+02 - Estimated value: 425.91 | Target value: 425.91
Example #4.259079e+02 - Estimated value: 425.91 | Target value: 425.91
Example #4.259097e+02 - Estimated value: 425.91 | Target value: 425.91
Example #4.259115e+02 - Estimated value: 425.91 | Target value: 425.91
Example #4.259133e+02 - Estimated value: 425.91 | Target value: 425.91
Example #4.259151e+02 - Estimated value: 425.92 | Target value: 425.92
Example #4.259168e+02 - Estimated value: 425.92 | Target value: 425.92
Example #4.259186e+02 - Estimated value: 425.92 | Target value: 425.92
Example #4.259204e+02 - Estimated value: 425.92 | Target value: 425.92
Example #4.259222e+02 - Estimated value: 425.92 | Target value: 425.92
Example #4.259240e+02 - Estimated value: 425.92 | Target value: 425.93
Example #4.259257e+02 - Estimated value: 425.93 | Target value: 425.93
Example #4.259275e+02 - Estimated value: 425.93 | Target value: 425.93
Example #4.259293e+02 - Estimated value: 425.93 | Target value: 425.93
Example #4.259311e+02 - Estimated value: 425.93 | Target value: 425.93
Example #4.259329e+02 - Estimated value: 425.93 | Target value: 425.93
Example #4.259346e+02 - Estimated value: 425.94 | Target value: 425.94
Example #4.259364e+02 - Estimated value: 425.94 | Target value: 425.94
Example #4.259382e+02 - Estimated value: 425.94 | Target value: 425.94
Example #4.259400e+02 - Estimated value: 425.94 | Target value: 425.94
Example #4.259417e+02 - Estimated value: 425.94 | Target value: 425.94
Example #4.259435e+02 - Estimated value: 425.94 | Target value: 425.94
Example #4.259453e+02 - Estimated value: 425.95 | Target value: 425.95
Example #4.259471e+02 - Estimated value: 425.95 | Target value: 425.95
Example #4.259489e+02 - Estimated value: 425.95 | Target value: 425.95
Example #4.259506e+02 - Estimated value: 425.95 | Target value: 425.95
Example #4.259524e+02 - Estimated value: 425.95 | Target value: 425.95
Example #4.259542e+02 - Estimated value: 425.95 | Target value: 425.96
Example #4.259560e+02 - Estimated value: 425.96 | Target value: 425.96
Example #4.259577e+02 - Estimated value: 425.96 | Target value: 425.96
Example #4.259595e+02 - Estimated value: 425.96 | Target value: 425.96
Example #4.259613e+02 - Estimated value: 425.96 | Target value: 425.96
Example #4.259631e+02 - Estimated value: 425.96 | Target value: 425.96
Example #4.259648e+02 - Estimated value: 425.97 | Target value: 425.97
Example #4.259666e+02 - Estimated value: 425.97 | Target value: 425.97
Example #4.259684e+02 - Estimated value: 425.97 | Target value: 425.97
Example #4.259701e+02 - Estimated value: 425.97 | Target value: 425.97
Example #4.259719e+02 - Estimated value: 425.97 | Target value: 425.97
Example #4.259737e+02 - Estimated value: 425.97 | Target value: 425.97
Example #4.259755e+02 - Estimated value: 425.98 | Target value: 425.98
Example #4.259772e+02 - Estimated value: 425.98 | Target value: 425.98
Example #4.259790e+02 - Estimated value: 425.98 | Target value: 425.98
Example #4.259808e+02 - Estimated value: 425.98 | Target value: 425.98
Example #4.259826e+02 - Estimated value: 425.98 | Target value: 425.98
Example #4.259843e+02 - Estimated value: 425.98 | Target value: 425.99
Example #4.259861e+02 - Estimated value: 425.99 | Target value: 425.99
Example #4.259879e+02 - Estimated value: 425.99 | Target value: 425.99
Example #4.259896e+02 - Estimated value: 425.99 | Target value: 425.99
Example #4.259914e+02 - Estimated value: 425.99 | Target value: 425.99
Example #4.259932e+02 - Estimated value: 425.99 | Target value: 425.99
Example #4.259949e+02 - Estimated value: 426.00 | Target value: 426.00
Example #4.259967e+02 - Estimated value: 426.00 | Target value: 426.00
Example #4.259985e+02 - Estimated value: 426.00 | Target value: 426.00
Example #4.260003e+02 - Estimated value: 426.00 | Target value: 426.00
Example #4.260020e+02 - Estimated value: 426.00 | Target value: 426.00
Example #4.260038e+02 - Estimated value: 426.00 | Target value: 426.00
Example #4.260056e+02 - Estimated value: 426.01 | Target value: 426.01
Example #4.260073e+02 - Estimated value: 426.01 | Target value: 426.01
Example #4.260091e+02 - Estimated value: 426.01 | Target value: 426.01
Example #4.260109e+02 - Estimated value: 426.01 | Target value: 426.01
Example #4.260126e+02 - Estimated value: 426.01 | Target value: 426.01
Example #4.260144e+02 - Estimated value: 426.01 | Target value: 426.02
Example #4.260162e+02 - Estimated value: 426.02 | Target value: 426.02
Example #4.260179e+02 - Estimated value: 426.02 | Target value: 426.02
Example #4.260197e+02 - Estimated value: 426.02 | Target value: 426.02
Example #4.260215e+02 - Estimated value: 426.02 | Target value: 426.02
Example #4.260232e+02 - Estimated value: 426.02 | Target value: 426.02
Example #4.260250e+02 - Estimated value: 426.03 | Target value: 426.03
Example #4.260268e+02 - Estimated value: 426.03 | Target value: 426.03
Example #4.260285e+02 - Estimated value: 426.03 | Target value: 426.03
Example #4.260303e+02 - Estimated value: 426.03 | Target value: 426.03
Example #4.260321e+02 - Estimated value: 426.03 | Target value: 426.03
Example #4.260338e+02 - Estimated value: 426.03 | Target value: 426.04
Example #4.260356e+02 - Estimated value: 426.04 | Target value: 426.04
Example #4.260374e+02 - Estimated value: 426.04 | Target value: 426.04
Example #4.260391e+02 - Estimated value: 426.04 | Target value: 426.04
Example #4.260409e+02 - Estimated value: 426.04 | Target value: 426.04
Example #4.260426e+02 - Estimated value: 426.04 | Target value: 426.04
Example #4.260444e+02 - Estimated value: 426.04 | Target value: 426.05
Example #4.260462e+02 - Estimated value: 426.05 | Target value: 426.05
Example #4.260479e+02 - Estimated value: 426.05 | Target value: 426.05
Example #4.260497e+02 - Estimated value: 426.05 | Target value: 426.05
Example #4.260515e+02 - Estimated value: 426.05 | Target value: 426.05
Example #4.260532e+02 - Estimated value: 426.05 | Target value: 426.05
Example #4.260550e+02 - Estimated value: 426.06 | Target value: 426.06
Example #4.260567e+02 - Estimated value: 426.06 | Target value: 426.06
Example #4.260585e+02 - Estimated value: 426.06 | Target value: 426.06
Example #4.260603e+02 - Estimated value: 426.06 | Target value: 426.06
Example #4.260620e+02 - Estimated value: 426.06 | Target value: 426.06
Example #4.260638e+02 - Estimated value: 426.06 | Target value: 426.06
Example #4.260656e+02 - Estimated value: 426.07 | Target value: 426.07
Example #4.260673e+02 - Estimated value: 426.07 | Target value: 426.07
Example #4.260691e+02 - Estimated value: 426.07 | Target value: 426.07
Example #4.260708e+02 - Estimated value: 426.07 | Target value: 426.07
Example #4.260726e+02 - Estimated value: 426.07 | Target value: 426.07
Example #4.260744e+02 - Estimated value: 426.07 | Target value: 426.08
Example #4.260761e+02 - Estimated value: 426.08 | Target value: 426.08
Example #4.260779e+02 - Estimated value: 426.08 | Target value: 426.08
Example #4.260796e+02 - Estimated value: 426.08 | Target value: 426.08
Example #4.260814e+02 - Estimated value: 426.08 | Target value: 426.08
Example #4.260831e+02 - Estimated value: 426.08 | Target value: 426.08
Example #4.260849e+02 - Estimated value: 426.09 | Target value: 426.09
Example #4.260867e+02 - Estimated value: 426.09 | Target value: 426.09
Example #4.260884e+02 - Estimated value: 426.09 | Target value: 426.09
Example #4.260902e+02 - Estimated value: 426.09 | Target value: 426.09
Example #4.260919e+02 - Estimated value: 426.09 | Target value: 426.09
Example #4.260937e+02 - Estimated value: 426.09 | Target value: 426.09
Example #4.260954e+02 - Estimated value: 426.10 | Target value: 426.10
Example #4.260972e+02 - Estimated value: 426.10 | Target value: 426.10
Example #4.260990e+02 - Estimated value: 426.10 | Target value: 426.10
Example #4.261007e+02 - Estimated value: 426.10 | Target value: 426.10
Example #4.261025e+02 - Estimated value: 426.10 | Target value: 426.10
Example #4.261042e+02 - Estimated value: 426.10 | Target value: 426.11
Example #4.261060e+02 - Estimated value: 426.11 | Target value: 426.11
Example #4.261077e+02 - Estimated value: 426.11 | Target value: 426.11
Example #4.261095e+02 - Estimated value: 426.11 | Target value: 426.11
Example #4.261112e+02 - Estimated value: 426.11 | Target value: 426.11
Example #4.261130e+02 - Estimated value: 426.11 | Target value: 426.11
Example #4.261148e+02 - Estimated value: 426.12 | Target value: 426.12
Example #4.261165e+02 - Estimated value: 426.12 | Target value: 426.12
Example #4.261183e+02 - Estimated value: 426.12 | Target value: 426.12
Example #4.261200e+02 - Estimated value: 426.12 | Target value: 426.12
Example #4.261218e+02 - Estimated value: 426.12 | Target value: 426.12
Example #4.261235e+02 - Estimated value: 426.12 | Target value: 426.12
Example #4.261253e+02 - Estimated value: 426.13 | Target value: 426.13
Example #4.261270e+02 - Estimated value: 426.13 | Target value: 426.13
Example #4.261288e+02 - Estimated value: 426.13 | Target value: 426.13
Example #4.261305e+02 - Estimated value: 426.13 | Target value: 426.13
Example #4.261323e+02 - Estimated value: 426.13 | Target value: 426.13
Example #4.261340e+02 - Estimated value: 426.13 | Target value: 426.14
Example #4.261358e+02 - Estimated value: 426.14 | Target value: 426.14
Example #4.261375e+02 - Estimated value: 426.14 | Target value: 426.14
Example #4.261393e+02 - Estimated value: 426.14 | Target value: 426.14
Example #4.261410e+02 - Estimated value: 426.14 | Target value: 426.14
Example #4.261428e+02 - Estimated value: 426.14 | Target value: 426.14
Example #4.261445e+02 - Estimated value: 426.15 | Target value: 426.15
Example #4.261463e+02 - Estimated value: 426.15 | Target value: 426.15
Example #4.261480e+02 - Estimated value: 426.15 | Target value: 426.15
Example #4.261498e+02 - Estimated value: 426.15 | Target value: 426.15
Example #4.261515e+02 - Estimated value: 426.15 | Target value: 426.15
Example #4.261533e+02 - Estimated value: 426.15 | Target value: 426.15
Example #4.261550e+02 - Estimated value: 426.16 | Target value: 426.16
Example #4.261568e+02 - Estimated value: 426.16 | Target value: 426.16
Example #4.261585e+02 - Estimated value: 426.16 | Target value: 426.16
Example #4.261603e+02 - Estimated value: 426.16 | Target value: 426.16
Example #4.261620e+02 - Estimated value: 426.16 | Target value: 426.16
Example #4.261638e+02 - Estimated value: 426.16 | Target value: 426.16
Example #4.261655e+02 - Estimated value: 426.17 | Target value: 426.17
Example #4.261673e+02 - Estimated value: 426.17 | Target value: 426.17
Example #4.261690e+02 - Estimated value: 426.17 | Target value: 426.17
Example #4.261708e+02 - Estimated value: 426.17 | Target value: 426.17
Example #4.261725e+02 - Estimated value: 426.17 | Target value: 426.17
Example #4.261743e+02 - Estimated value: 426.17 | Target value: 426.18
Example #4.261760e+02 - Estimated value: 426.18 | Target value: 426.18
Example #4.261777e+02 - Estimated value: 426.18 | Target value: 426.18
Example #4.261795e+02 - Estimated value: 426.18 | Target value: 426.18
Example #4.261812e+02 - Estimated value: 426.18 | Target value: 426.18
Example #4.261830e+02 - Estimated value: 426.18 | Target value: 426.18
Example #4.261847e+02 - Estimated value: 426.19 | Target value: 460.00
Example #2 - Estimated value: 341.97 | Target value: 275.72
Example #2.885561e+02 - Estimated value: 286.07 | Target value: 286.55
Example #2.864569e+02 - Estimated value: 286.47 | Target value: 286.47
Example #2.864713e+02 - Estimated value: 286.47 | Target value: 286.47
Example #2.864702e+02 - Estimated value: 286.47 | Target value: 286.47
Example #2.864693e+02 - Estimated value: 286.47 | Target value: 286.47
Example #2.864683e+02 - Estimated value: 286.47 | Target value: 286.47
Example #2.864674e+02 - Estimated value: 286.47 | Target value: 286.47
Example #2.864665e+02 - Estimated value: 286.47 | Target value: 286.47
Example #2.864655e+02 - Estimated value: 286.47 | Target value: 286.46
Example #2.864646e+02 - Estimated value: 286.46 | Target value: 286.46
Example #2.864636e+02 - Estimated value: 286.46 | Target value: 286.46
Example #2.864627e+02 - Estimated value: 286.46 | Target value: 286.46
Example #2.864618e+02 - Estimated value: 286.46 | Target value: 286.46
Example #2.864608e+02 - Estimated value: 286.46 | Target value: 286.46
Example #2.864599e+02 - Estimated value: 286.46 | Target value: 286.46
Example #2.864590e+02 - Estimated value: 286.46 | Target value: 286.46
Example #2.864580e+02 - Estimated value: 286.46 | Target value: 286.46
Example #2.864571e+02 - Estimated value: 286.46 | Target value: 286.46
Example #2.864561e+02 - Estimated value: 286.46 | Target value: 286.46
Example #2.864552e+02 - Estimated value: 286.45 | Target value: 286.45
Example #2.864543e+02 - Estimated value: 286.45 | Target value: 286.45
Example #2.864533e+02 - Estimated value: 286.45 | Target value: 286.45
Example #2.864524e+02 - Estimated value: 286.45 | Target value: 286.45
Example #2.864515e+02 - Estimated value: 286.45 | Target value: 286.45
Example #2.864505e+02 - Estimated value: 286.45 | Target value: 286.45
Example #2.864496e+02 - Estimated value: 286.45 | Target value: 286.45
Example #2.864486e+02 - Estimated value: 286.45 | Target value: 286.45
Example #2.864477e+02 - Estimated value: 286.45 | Target value: 286.45
Example #2.864468e+02 - Estimated value: 286.45 | Target value: 286.45
Example #2.864458e+02 - Estimated value: 286.45 | Target value: 286.45
Example #2.864449e+02 - Estimated value: 286.44 | Target value: 286.44
Example #2.864440e+02 - Estimated value: 286.44 | Target value: 286.44
Example #2.864430e+02 - Estimated value: 286.44 | Target value: 286.44
Example #2.864421e+02 - Estimated value: 286.44 | Target value: 286.44
Example #2.864412e+02 - Estimated value: 286.44 | Target value: 286.44
Example #2.864402e+02 - Estimated value: 286.44 | Target value: 286.44
Example #2.864393e+02 - Estimated value: 286.44 | Target value: 286.44
Example #2.864383e+02 - Estimated value: 286.44 | Target value: 286.44
Example #2.864374e+02 - Estimated value: 286.44 | Target value: 286.44
Example #2.864365e+02 - Estimated value: 286.44 | Target value: 286.44
Example #2.864355e+02 - Estimated value: 286.44 | Target value: 286.43
Example #2.864346e+02 - Estimated value: 286.43 | Target value: 286.43
Example #2.864337e+02 - Estimated value: 286.43 | Target value: 286.43
Example #2.864327e+02 - Estimated value: 286.43 | Target value: 286.43
Example #2.864318e+02 - Estimated value: 286.43 | Target value: 286.43
Example #2.864309e+02 - Estimated value: 286.43 | Target value: 286.43
Example #2.864299e+02 - Estimated value: 286.43 | Target value: 286.43
Example #2.864290e+02 - Estimated value: 286.43 | Target value: 286.43
Example #2.864281e+02 - Estimated value: 286.43 | Target value: 286.43
Example #2.864271e+02 - Estimated value: 286.43 | Target value: 286.43
Example #2.864262e+02 - Estimated value: 286.43 | Target value: 286.43
Example #2.864253e+02 - Estimated value: 286.42 | Target value: 286.42
Example #2.864243e+02 - Estimated value: 286.42 | Target value: 286.42
Example #2.864234e+02 - Estimated value: 286.42 | Target value: 286.42
Example #2.864225e+02 - Estimated value: 286.42 | Target value: 286.42
Example #2.864215e+02 - Estimated value: 286.42 | Target value: 286.42
Example #2.864206e+02 - Estimated value: 286.42 | Target value: 286.42
Example #2.864197e+02 - Estimated value: 286.42 | Target value: 286.42
Example #2.864187e+02 - Estimated value: 286.42 | Target value: 286.42
Example #2.864178e+02 - Estimated value: 286.42 | Target value: 286.42
Example #2.864169e+02 - Estimated value: 286.42 | Target value: 286.42
Example #2.864159e+02 - Estimated value: 286.42 | Target value: 286.42
Example #2.864150e+02 - Estimated value: 286.41 | Target value: 286.41
Example #2.864141e+02 - Estimated value: 286.41 | Target value: 286.41
Example #2.864131e+02 - Estimated value: 286.41 | Target value: 286.41
Example #2.864122e+02 - Estimated value: 286.41 | Target value: 286.41
Example #2.864113e+02 - Estimated value: 286.41 | Target value: 286.41
Example #2.864104e+02 - Estimated value: 286.41 | Target value: 286.41
Example #2.864094e+02 - Estimated value: 286.41 | Target value: 286.41
Example #2.864085e+02 - Estimated value: 286.41 | Target value: 286.41
Example #2.864076e+02 - Estimated value: 286.41 | Target value: 286.41
Example #2.864066e+02 - Estimated value: 286.41 | Target value: 286.41
Example #2.864057e+02 - Estimated value: 286.41 | Target value: 286.41
Example #2.864048e+02 - Estimated value: 286.40 | Target value: 286.40
Example #2.864038e+02 - Estimated value: 286.40 | Target value: 286.40
Example #2.864029e+02 - Estimated value: 286.40 | Target value: 286.40
Example #2.864020e+02 - Estimated value: 286.40 | Target value: 286.40
Example #2.864010e+02 - Estimated value: 286.40 | Target value: 286.40
Example #2.864001e+02 - Estimated value: 286.40 | Target value: 286.40
Example #2.863992e+02 - Estimated value: 286.40 | Target value: 286.40
Example #2.863983e+02 - Estimated value: 286.40 | Target value: 286.40
Example #2.863973e+02 - Estimated value: 286.40 | Target value: 286.40
Example #2.863964e+02 - Estimated value: 286.40 | Target value: 286.40
Example #2.863955e+02 - Estimated value: 286.40 | Target value: 286.39
Example #2.863945e+02 - Estimated value: 286.39 | Target value: 286.39
Example #2.863936e+02 - Estimated value: 286.39 | Target value: 286.39
Example #2.863927e+02 - Estimated value: 286.39 | Target value: 286.39
Example #2.863918e+02 - Estimated value: 286.39 | Target value: 286.39
Example #2.863908e+02 - Estimated value: 286.39 | Target value: 286.39
Example #2.863899e+02 - Estimated value: 286.39 | Target value: 286.39
Example #2.863890e+02 - Estimated value: 286.39 | Target value: 286.39
Example #2.863880e+02 - Estimated value: 286.39 | Target value: 286.39
Example #2.863871e+02 - Estimated value: 286.39 | Target value: 286.39
Example #2.863862e+02 - Estimated value: 286.39 | Target value: 286.39
Example #2.863853e+02 - Estimated value: 286.38 | Target value: 286.38
Example #2.863843e+02 - Estimated value: 286.38 | Target value: 286.38
Example #2.863834e+02 - Estimated value: 286.38 | Target value: 286.38
Example #2.863825e+02 - Estimated value: 286.38 | Target value: 286.38
Example #2.863815e+02 - Estimated value: 286.38 | Target value: 286.38
Example #2.863806e+02 - Estimated value: 286.38 | Target value: 286.38
Example #2.863797e+02 - Estimated value: 286.38 | Target value: 286.38
Example #2.863788e+02 - Estimated value: 286.38 | Target value: 286.38
Example #2.863778e+02 - Estimated value: 286.38 | Target value: 286.38
Example #2.863769e+02 - Estimated value: 286.38 | Target value: 286.38
Example #2.863760e+02 - Estimated value: 286.38 | Target value: 286.38
Example #2.863750e+02 - Estimated value: 286.37 | Target value: 286.37
Example #2.863741e+02 - Estimated value: 286.37 | Target value: 286.37
Example #2.863732e+02 - Estimated value: 286.37 | Target value: 286.37
Example #2.863723e+02 - Estimated value: 286.37 | Target value: 286.37
Example #2.863713e+02 - Estimated value: 286.37 | Target value: 286.37
Example #2.863704e+02 - Estimated value: 286.37 | Target value: 286.37
Example #2.863695e+02 - Estimated value: 286.37 | Target value: 286.37
Example #2.863686e+02 - Estimated value: 286.37 | Target value: 286.37
Example #2.863676e+02 - Estimated value: 286.37 | Target value: 286.37
Example #2.863667e+02 - Estimated value: 286.37 | Target value: 286.37
Example #2.863658e+02 - Estimated value: 286.37 | Target value: 286.37
Example #2.863649e+02 - Estimated value: 286.36 | Target value: 286.36
Example #2.863639e+02 - Estimated value: 286.36 | Target value: 286.36
Example #2.863630e+02 - Estimated value: 286.36 | Target value: 286.36
Example #2.863621e+02 - Estimated value: 286.36 | Target value: 286.36
Example #2.863612e+02 - Estimated value: 286.36 | Target value: 286.36
Example #2.863602e+02 - Estimated value: 286.36 | Target value: 286.36
Example #2.863593e+02 - Estimated value: 286.36 | Target value: 286.36
Example #2.863584e+02 - Estimated value: 286.36 | Target value: 286.36
Example #2.863575e+02 - Estimated value: 286.36 | Target value: 286.36
Example #2.863565e+02 - Estimated value: 286.36 | Target value: 286.36
Example #2.863556e+02 - Estimated value: 286.36 | Target value: 286.35
Example #2.863547e+02 - Estimated value: 286.35 | Target value: 286.35
Example #2.863538e+02 - Estimated value: 286.35 | Target value: 286.35
Example #2.863528e+02 - Estimated value: 286.35 | Target value: 286.35
Example #2.863519e+02 - Estimated value: 286.35 | Target value: 286.35
Example #2.863510e+02 - Estimated value: 286.35 | Target value: 286.35
Example #2.863501e+02 - Estimated value: 286.35 | Target value: 286.35
Example #2.863491e+02 - Estimated value: 286.35 | Target value: 286.35
Example #2.863482e+02 - Estimated value: 286.35 | Target value: 286.35
Example #2.863473e+02 - Estimated value: 286.35 | Target value: 286.35
Example #2.863464e+02 - Estimated value: 286.35 | Target value: 286.35
Example #2.863455e+02 - Estimated value: 286.35 | Target value: 286.34
Example #2.863445e+02 - Estimated value: 286.34 | Target value: 286.34
Example #2.863436e+02 - Estimated value: 286.34 | Target value: 286.34
Example #2.863427e+02 - Estimated value: 286.34 | Target value: 286.34
Example #2.863418e+02 - Estimated value: 286.34 | Target value: 286.34
Example #2.863408e+02 - Estimated value: 286.34 | Target value: 286.34
Example #2.863399e+02 - Estimated value: 286.34 | Target value: 286.34
Example #2.863390e+02 - Estimated value: 286.34 | Target value: 286.34
Example #2.863381e+02 - Estimated value: 286.34 | Target value: 286.34
Example #2.863372e+02 - Estimated value: 286.34 | Target value: 286.34
Example #2.863362e+02 - Estimated value: 286.34 | Target value: 286.34
Example #2.863353e+02 - Estimated value: 286.34 | Target value: 286.33
Example #2.863344e+02 - Estimated value: 286.33 | Target value: 286.33
Example #2.863335e+02 - Estimated value: 286.33 | Target value: 286.33
Example #2.863325e+02 - Estimated value: 286.33 | Target value: 286.33
Example #2.863316e+02 - Estimated value: 286.33 | Target value: 286.33
Example #2.863307e+02 - Estimated value: 286.33 | Target value: 286.33
Example #2.863298e+02 - Estimated value: 286.33 | Target value: 286.33
Example #2.863289e+02 - Estimated value: 286.33 | Target value: 286.33
Example #2.863279e+02 - Estimated value: 286.33 | Target value: 286.33
Example #2.863270e+02 - Estimated value: 286.33 | Target value: 286.33
Example #2.863261e+02 - Estimated value: 286.33 | Target value: 286.33
Example #2.863252e+02 - Estimated value: 286.32 | Target value: 286.32
Example #2.863243e+02 - Estimated value: 286.32 | Target value: 286.32
Example #2.863233e+02 - Estimated value: 286.32 | Target value: 286.32
Example #2.863224e+02 - Estimated value: 286.32 | Target value: 286.32
Example #2.863215e+02 - Estimated value: 286.32 | Target value: 286.32
Example #2.863206e+02 - Estimated value: 286.32 | Target value: 286.32
Example #2.863197e+02 - Estimated value: 286.32 | Target value: 286.32
Example #2.863187e+02 - Estimated value: 286.32 | Target value: 286.32
Example #2.863178e+02 - Estimated value: 286.32 | Target value: 286.32
Example #2.863169e+02 - Estimated value: 286.32 | Target value: 286.32
Example #2.863160e+02 - Estimated value: 286.32 | Target value: 286.32
Example #2.863151e+02 - Estimated value: 286.31 | Target value: 286.31
Example #2.863141e+02 - Estimated value: 286.31 | Target value: 286.31
Example #2.863132e+02 - Estimated value: 286.31 | Target value: 286.31
Example #2.863123e+02 - Estimated value: 286.31 | Target value: 286.31
Example #2.863114e+02 - Estimated value: 286.31 | Target value: 286.31
Example #2.863105e+02 - Estimated value: 286.31 | Target value: 286.31
Example #2.863096e+02 - Estimated value: 286.31 | Target value: 286.31
Example #2.863086e+02 - Estimated value: 286.31 | Target value: 286.31
Example #2.863077e+02 - Estimated value: 286.31 | Target value: 286.31
Example #2.863068e+02 - Estimated value: 286.31 | Target value: 286.31
Example #2.863059e+02 - Estimated value: 286.31 | Target value: 286.31
Example #2.863050e+02 - Estimated value: 286.30 | Target value: 286.30
Example #2.863040e+02 - Estimated value: 286.30 | Target value: 286.30
Example #2.863031e+02 - Estimated value: 286.30 | Target value: 286.30
Example #2.863022e+02 - Estimated value: 286.30 | Target value: 286.30
Example #2.863013e+02 - Estimated value: 286.30 | Target value: 286.30
Example #2.863004e+02 - Estimated value: 286.30 | Target value: 286.30
Example #2.862995e+02 - Estimated value: 286.30 | Target value: 286.30
Example #2.862985e+02 - Estimated value: 286.30 | Target value: 286.30
Example #2.862976e+02 - Estimated value: 286.30 | Target value: 286.30
Example #2.862967e+02 - Estimated value: 286.30 | Target value: 286.30
Example #2.862958e+02 - Estimated value: 286.30 | Target value: 286.30
Example #2.862949e+02 - Estimated value: 286.29 | Target value: 286.29
Example #2.862940e+02 - Estimated value: 286.29 | Target value: 286.29
Example #2.862930e+02 - Estimated value: 286.29 | Target value: 286.29
Example #2.862921e+02 - Estimated value: 286.29 | Target value: 286.29
Example #2.862912e+02 - Estimated value: 286.29 | Target value: 286.29
Example #2.862903e+02 - Estimated value: 286.29 | Target value: 286.29
Example #2.862894e+02 - Estimated value: 286.29 | Target value: 286.29
Example #2.862885e+02 - Estimated value: 286.29 | Target value: 286.29
Example #2.862876e+02 - Estimated value: 286.29 | Target value: 286.29
Example #2.862866e+02 - Estimated value: 286.29 | Target value: 286.29
Example #2.862857e+02 - Estimated value: 286.29 | Target value: 286.29
Example #2.862848e+02 - Estimated value: 286.28 | Target value: 286.28
Example #2.862839e+02 - Estimated value: 286.28 | Target value: 286.28
Example #2.862830e+02 - Estimated value: 286.28 | Target value: 286.28
Example #2.862821e+02 - Estimated value: 286.28 | Target value: 286.28
Example #2.862811e+02 - Estimated value: 286.28 | Target value: 286.28
Example #2.862802e+02 - Estimated value: 286.28 | Target value: 286.28
Example #2.862793e+02 - Estimated value: 286.28 | Target value: 286.28
Example #2.862784e+02 - Estimated value: 286.28 | Target value: 286.28
Example #2.862775e+02 - Estimated value: 286.28 | Target value: 286.28
Example #2.862766e+02 - Estimated value: 286.28 | Target value: 286.28
Example #2.862757e+02 - Estimated value: 286.28 | Target value: 286.28
Example #2.862748e+02 - Estimated value: 286.27 | Target value: 286.27
Example #2.862738e+02 - Estimated value: 286.27 | Target value: 286.27
Example #2.862729e+02 - Estimated value: 286.27 | Target value: 286.27
Example #2.862720e+02 - Estimated value: 286.27 | Target value: 286.27
Example #2.862711e+02 - Estimated value: 286.27 | Target value: 286.27
Example #2.862702e+02 - Estimated value: 286.27 | Target value: 286.27
Example #2.862693e+02 - Estimated value: 286.27 | Target value: 286.27
Example #2.862684e+02 - Estimated value: 286.27 | Target value: 286.27
Example #2.862674e+02 - Estimated value: 286.27 | Target value: 286.27
Example #2.862665e+02 - Estimated value: 286.27 | Target value: 286.27
Example #2.862656e+02 - Estimated value: 286.27 | Target value: 286.27
Example #2.862647e+02 - Estimated value: 286.26 | Target value: 286.26
Example #2.862638e+02 - Estimated value: 286.26 | Target value: 286.26
Example #2.862629e+02 - Estimated value: 286.26 | Target value: 286.26
Example #2.862620e+02 - Estimated value: 286.26 | Target value: 286.26
Example #2.862611e+02 - Estimated value: 286.26 | Target value: 286.26
Example #2.862601e+02 - Estimated value: 286.26 | Target value: 286.26
Example #2.862592e+02 - Estimated value: 286.26 | Target value: 286.26
Example #2.862583e+02 - Estimated value: 286.26 | Target value: 286.26
Example #2.862574e+02 - Estimated value: 286.26 | Target value: 286.26
Example #2.862565e+02 - Estimated value: 286.26 | Target value: 286.26
Example #2.862556e+02 - Estimated value: 286.26 | Target value: 286.25
Example #2.862547e+02 - Estimated value: 286.25 | Target value: 286.25
Example #2.862538e+02 - Estimated value: 286.25 | Target value: 286.25
Example #2.862529e+02 - Estimated value: 286.25 | Target value: 286.25
Example #2.862519e+02 - Estimated value: 286.25 | Target value: 286.25
Example #2.862510e+02 - Estimated value: 286.25 | Target value: 286.25
Example #2.862501e+02 - Estimated value: 286.25 | Target value: 286.25
Example #2.862492e+02 - Estimated value: 286.25 | Target value: 286.25
Example #2.862483e+02 - Estimated value: 286.25 | Target value: 286.25
Example #2.862474e+02 - Estimated value: 286.25 | Target value: 286.25
Example #2.862465e+02 - Estimated value: 286.25 | Target value: 286.25
Example #2.862456e+02 - Estimated value: 286.25 | Target value: 286.24
Example #2.862447e+02 - Estimated value: 286.24 | Target value: 286.24
Example #2.862438e+02 - Estimated value: 286.24 | Target value: 286.24
Example #2.862428e+02 - Estimated value: 286.24 | Target value: 286.24
Example #2.862419e+02 - Estimated value: 286.24 | Target value: 286.24
Example #2.862410e+02 - Estimated value: 286.24 | Target value: 286.24
Example #2.862401e+02 - Estimated value: 286.24 | Target value: 286.24
Example #2.862392e+02 - Estimated value: 286.24 | Target value: 286.24
Example #2.862383e+02 - Estimated value: 286.24 | Target value: 286.24
Example #2.862374e+02 - Estimated value: 286.24 | Target value: 286.24
Example #2.862365e+02 - Estimated value: 286.24 | Target value: 286.24
Example #2.862356e+02 - Estimated value: 286.24 | Target value: 286.23
Example #2.862347e+02 - Estimated value: 286.23 | Target value: 286.23
Example #2.862338e+02 - Estimated value: 286.23 | Target value: 286.23
Example #2.862329e+02 - Estimated value: 286.23 | Target value: 286.23
Example #2.862319e+02 - Estimated value: 286.23 | Target value: 286.23
Example #2.862310e+02 - Estimated value: 286.23 | Target value: 286.23
Example #2.862301e+02 - Estimated value: 286.23 | Target value: 286.23
Example #2.862292e+02 - Estimated value: 286.23 | Target value: 286.23
Example #2.862283e+02 - Estimated value: 286.23 | Target value: 286.23
Example #2.862274e+02 - Estimated value: 286.23 | Target value: 286.23
Example #2.862265e+02 - Estimated value: 286.23 | Target value: 286.23
Example #2.862256e+02 - Estimated value: 286.23 | Target value: 286.22
Example #2.862247e+02 - Estimated value: 286.22 | Target value: 286.22
Example #2.862238e+02 - Estimated value: 286.22 | Target value: 286.22
Example #2.862229e+02 - Estimated value: 286.22 | Target value: 286.22
Example #2.862220e+02 - Estimated value: 286.22 | Target value: 286.22
Example #2.862211e+02 - Estimated value: 286.22 | Target value: 286.22
Example #2.862201e+02 - Estimated value: 286.22 | Target value: 286.22
Example #2.862192e+02 - Estimated value: 286.22 | Target value: 286.22
Example #2.862183e+02 - Estimated value: 286.22 | Target value: 286.22
Example #2.862174e+02 - Estimated value: 286.22 | Target value: 286.22
Example #2.862165e+02 - Estimated value: 286.22 | Target value: 286.22
Example #2.862156e+02 - Estimated value: 286.22 | Target value: 286.22
Example #2.862147e+02 - Estimated value: 286.21 | Target value: 286.21
Example #2.862138e+02 - Estimated value: 286.21 | Target value: 286.21
Example #2.862129e+02 - Estimated value: 286.21 | Target value: 286.21
Example #2.862120e+02 - Estimated value: 286.21 | Target value: 286.21
Example #2.862111e+02 - Estimated value: 286.21 | Target value: 286.21
Example #2.862102e+02 - Estimated value: 286.21 | Target value: 286.21
Example #2.862093e+02 - Estimated value: 286.21 | Target value: 286.21
Example #2.862084e+02 - Estimated value: 286.21 | Target value: 286.21
Example #2.862075e+02 - Estimated value: 286.21 | Target value: 286.21
Example #2.862066e+02 - Estimated value: 286.21 | Target value: 286.21
Example #2.862057e+02 - Estimated value: 286.21 | Target value: 286.21
Example #2.862048e+02 - Estimated value: 286.20 | Target value: 286.20
Example #2.862039e+02 - Estimated value: 286.20 | Target value: 286.20
Example #2.862029e+02 - Estimated value: 286.20 | Target value: 286.20
Example #2.862020e+02 - Estimated value: 286.20 | Target value: 286.20
Example #2.862011e+02 - Estimated value: 286.20 | Target value: 286.20
Example #2.862002e+02 - Estimated value: 286.20 | Target value: 286.20
Example #2.861993e+02 - Estimated value: 286.20 | Target value: 286.20
Example #2.861984e+02 - Estimated value: 286.20 | Target value: 286.20
Example #2.861975e+02 - Estimated value: 286.20 | Target value: 286.20
Example #2.861966e+02 - Estimated value: 286.20 | Target value: 286.20
Example #2.861957e+02 - Estimated value: 286.20 | Target value: 286.20
Example #2.861948e+02 - Estimated value: 286.19 | Target value: 286.19
Example #2.861939e+02 - Estimated value: 286.19 | Target value: 286.19
Example #2.861930e+02 - Estimated value: 286.19 | Target value: 286.19
Example #2.861921e+02 - Estimated value: 286.19 | Target value: 286.19
Example #2.861912e+02 - Estimated value: 286.19 | Target value: 286.19
Example #2.861903e+02 - Estimated value: 286.19 | Target value: 286.19
Example #2.861894e+02 - Estimated value: 286.19 | Target value: 286.19
Example #2.861885e+02 - Estimated value: 286.19 | Target value: 286.19
Example #2.861876e+02 - Estimated value: 286.19 | Target value: 286.19
Example #2.861867e+02 - Estimated value: 286.19 | Target value: 286.19
Example #2.861858e+02 - Estimated value: 286.19 | Target value: 286.19
Example #2.861849e+02 - Estimated value: 286.18 | Target value: 286.18
Example #2.861840e+02 - Estimated value: 286.18 | Target value: 286.18
Example #2.861831e+02 - Estimated value: 286.18 | Target value: 286.18
Example #2.861822e+02 - Estimated value: 286.18 | Target value: 286.18
Example #2.861813e+02 - Estimated value: 286.18 | Target value: 286.18
Example #2.861804e+02 - Estimated value: 286.18 | Target value: 286.18
Example #2.861795e+02 - Estimated value: 286.18 | Target value: 286.18
Example #2.861786e+02 - Estimated value: 286.18 | Target value: 286.18
Example #2.861777e+02 - Estimated value: 286.18 | Target value: 286.18
Example #2.861768e+02 - Estimated value: 286.18 | Target value: 286.18
Example #2.861759e+02 - Estimated value: 286.18 | Target value: 286.18
Example #2.861750e+02 - Estimated value: 286.17 | Target value: 286.17
Example #2.861741e+02 - Estimated value: 286.17 | Target value: 286.17
Example #2.861732e+02 - Estimated value: 286.17 | Target value: 286.17
Example #2.861723e+02 - Estimated value: 286.17 | Target value: 286.17
Example #2.861714e+02 - Estimated value: 286.17 | Target value: 286.17
Example #2.861705e+02 - Estimated value: 286.17 | Target value: 286.17
Example #2.861696e+02 - Estimated value: 286.17 | Target value: 286.17
Example #2.861687e+02 - Estimated value: 286.17 | Target value: 286.17
Example #2.861678e+02 - Estimated value: 286.17 | Target value: 232.00
Example #3 - Estimated value: 205.83 | Target value: 165.95
Example #1.736768e+02 - Estimated value: 172.18 | Target value: 172.47
Example #1.724110e+02 - Estimated value: 172.42 | Target value: 172.42
Example #1.724172e+02 - Estimated value: 172.42 | Target value: 172.42
Example #1.724142e+02 - Estimated value: 172.41 | Target value: 172.41
Example #1.724113e+02 - Estimated value: 172.41 | Target value: 172.41
Example #1.724083e+02 - Estimated value: 172.41 | Target value: 172.41
Example #1.724054e+02 - Estimated value: 172.40 | Target value: 172.40
Example #1.724025e+02 - Estimated value: 172.40 | Target value: 172.40
Example #1.723995e+02 - Estimated value: 172.40 | Target value: 172.40
Example #1.723966e+02 - Estimated value: 172.40 | Target value: 172.39
Example #1.723937e+02 - Estimated value: 172.39 | Target value: 172.39
Example #1.723907e+02 - Estimated value: 172.39 | Target value: 172.39
Example #1.723878e+02 - Estimated value: 172.39 | Target value: 172.39
Example #1.723849e+02 - Estimated value: 172.38 | Target value: 172.38
Example #1.723819e+02 - Estimated value: 172.38 | Target value: 172.38
Example #1.723790e+02 - Estimated value: 172.38 | Target value: 172.38
Example #1.723761e+02 - Estimated value: 172.38 | Target value: 172.37
Example #1.723731e+02 - Estimated value: 172.37 | Target value: 172.37
Example #1.723702e+02 - Estimated value: 172.37 | Target value: 172.37
Example #1.723673e+02 - Estimated value: 172.37 | Target value: 172.37
Example #1.723643e+02 - Estimated value: 172.36 | Target value: 172.36
Example #1.723614e+02 - Estimated value: 172.36 | Target value: 172.36
Example #1.723585e+02 - Estimated value: 172.36 | Target value: 172.36
Example #1.723555e+02 - Estimated value: 172.35 | Target value: 172.35
Example #1.723526e+02 - Estimated value: 172.35 | Target value: 172.35
Example #1.723497e+02 - Estimated value: 172.35 | Target value: 172.35
Example #1.723467e+02 - Estimated value: 172.35 | Target value: 172.34
Example #1.723438e+02 - Estimated value: 172.34 | Target value: 172.34
Example #1.723409e+02 - Estimated value: 172.34 | Target value: 172.34
Example #1.723380e+02 - Estimated value: 172.34 | Target value: 172.34
Example #1.723350e+02 - Estimated value: 172.33 | Target value: 172.33
Example #1.723321e+02 - Estimated value: 172.33 | Target value: 172.33
Example #1.723292e+02 - Estimated value: 172.33 | Target value: 172.33
Example #1.723263e+02 - Estimated value: 172.33 | Target value: 172.32
Example #1.723233e+02 - Estimated value: 172.32 | Target value: 172.32
Example #1.723204e+02 - Estimated value: 172.32 | Target value: 172.32
Example #1.723175e+02 - Estimated value: 172.32 | Target value: 172.32
Example #1.723145e+02 - Estimated value: 172.31 | Target value: 172.31
Example #1.723116e+02 - Estimated value: 172.31 | Target value: 172.31
Example #1.723087e+02 - Estimated value: 172.31 | Target value: 172.31
Example #1.723058e+02 - Estimated value: 172.30 | Target value: 172.30
Example #1.723029e+02 - Estimated value: 172.30 | Target value: 172.30
Example #1.722999e+02 - Estimated value: 172.30 | Target value: 172.30
Example #1.722970e+02 - Estimated value: 172.30 | Target value: 172.30
Example #1.722941e+02 - Estimated value: 172.29 | Target value: 172.29
Example #1.722912e+02 - Estimated value: 172.29 | Target value: 172.29
Example #1.722882e+02 - Estimated value: 172.29 | Target value: 172.29
Example #1.722853e+02 - Estimated value: 172.28 | Target value: 172.28
Example #1.722824e+02 - Estimated value: 172.28 | Target value: 172.28
Example #1.722795e+02 - Estimated value: 172.28 | Target value: 172.28
Example #1.722766e+02 - Estimated value: 172.28 | Target value: 172.27
Example #1.722736e+02 - Estimated value: 172.27 | Target value: 172.27
Example #1.722707e+02 - Estimated value: 172.27 | Target value: 172.27
Example #1.722678e+02 - Estimated value: 172.27 | Target value: 172.27
Example #1.722649e+02 - Estimated value: 172.26 | Target value: 172.26
Example #1.722620e+02 - Estimated value: 172.26 | Target value: 172.26
Example #1.722590e+02 - Estimated value: 172.26 | Target value: 172.26
Example #1.722561e+02 - Estimated value: 172.26 | Target value: 172.25
Example #1.722532e+02 - Estimated value: 172.25 | Target value: 172.25
Example #1.722503e+02 - Estimated value: 172.25 | Target value: 172.25
Example #1.722474e+02 - Estimated value: 172.25 | Target value: 172.25
Example #1.722445e+02 - Estimated value: 172.24 | Target value: 172.24
Example #1.722416e+02 - Estimated value: 172.24 | Target value: 172.24
Example #1.722386e+02 - Estimated value: 172.24 | Target value: 172.24
Example #1.722357e+02 - Estimated value: 172.23 | Target value: 172.23
Example #1.722328e+02 - Estimated value: 172.23 | Target value: 172.23
Example #1.722299e+02 - Estimated value: 172.23 | Target value: 172.23
Example #1.722270e+02 - Estimated value: 172.23 | Target value: 172.23
Example #1.722241e+02 - Estimated value: 172.22 | Target value: 172.22
Example #1.722212e+02 - Estimated value: 172.22 | Target value: 172.22
Example #1.722182e+02 - Estimated value: 172.22 | Target value: 172.22
Example #1.722153e+02 - Estimated value: 172.21 | Target value: 172.21
Example #1.722124e+02 - Estimated value: 172.21 | Target value: 172.21
Example #1.722095e+02 - Estimated value: 172.21 | Target value: 172.21
Example #1.722066e+02 - Estimated value: 172.21 | Target value: 172.20
Example #1.722037e+02 - Estimated value: 172.20 | Target value: 172.20
Example #1.722008e+02 - Estimated value: 172.20 | Target value: 172.20
Example #1.721979e+02 - Estimated value: 172.20 | Target value: 172.20
Example #1.721950e+02 - Estimated value: 172.19 | Target value: 172.19
Example #1.721921e+02 - Estimated value: 172.19 | Target value: 172.19
Example #1.721891e+02 - Estimated value: 172.19 | Target value: 172.19
Example #1.721862e+02 - Estimated value: 172.19 | Target value: 172.18
Example #1.721833e+02 - Estimated value: 172.18 | Target value: 172.18
Example #1.721804e+02 - Estimated value: 172.18 | Target value: 172.18
Example #1.721775e+02 - Estimated value: 172.18 | Target value: 172.18
Example #1.721746e+02 - Estimated value: 172.17 | Target value: 172.17
Example #1.721717e+02 - Estimated value: 172.17 | Target value: 172.17
Example #1.721688e+02 - Estimated value: 172.17 | Target value: 172.17
Example #1.721659e+02 - Estimated value: 172.16 | Target value: 172.16
Example #1.721630e+02 - Estimated value: 172.16 | Target value: 172.16
Example #1.721601e+02 - Estimated value: 172.16 | Target value: 172.16
Example #1.721572e+02 - Estimated value: 172.16 | Target value: 172.16
Example #1.721543e+02 - Estimated value: 172.15 | Target value: 172.15
Example #1.721514e+02 - Estimated value: 172.15 | Target value: 172.15
Example #1.721485e+02 - Estimated value: 172.15 | Target value: 172.15
Example #1.721456e+02 - Estimated value: 172.14 | Target value: 172.14
Example #1.721427e+02 - Estimated value: 172.14 | Target value: 172.14
Example #1.721398e+02 - Estimated value: 172.14 | Target value: 172.14
Example #1.721369e+02 - Estimated value: 172.14 | Target value: 172.13
Example #1.721340e+02 - Estimated value: 172.13 | Target value: 172.13
Example #1.721311e+02 - Estimated value: 172.13 | Target value: 172.13
Example #1.721282e+02 - Estimated value: 172.13 | Target value: 172.13
Example #1.721253e+02 - Estimated value: 172.12 | Target value: 172.12
Example #1.721224e+02 - Estimated value: 172.12 | Target value: 172.12
Example #1.721195e+02 - Estimated value: 172.12 | Target value: 172.12
Example #1.721166e+02 - Estimated value: 172.12 | Target value: 172.11
Example #1.721137e+02 - Estimated value: 172.11 | Target value: 172.11
Example #1.721108e+02 - Estimated value: 172.11 | Target value: 172.11
Example #1.721079e+02 - Estimated value: 172.11 | Target value: 172.11
Example #1.721050e+02 - Estimated value: 172.10 | Target value: 172.10
Example #1.721021e+02 - Estimated value: 172.10 | Target value: 172.10
Example #1.720992e+02 - Estimated value: 172.10 | Target value: 172.10
Example #1.720963e+02 - Estimated value: 172.10 | Target value: 172.09
Example #1.720934e+02 - Estimated value: 172.09 | Target value: 172.09
Example #1.720905e+02 - Estimated value: 172.09 | Target value: 172.09
Example #1.720876e+02 - Estimated value: 172.09 | Target value: 172.09
Example #1.720847e+02 - Estimated value: 172.08 | Target value: 172.08
Example #1.720818e+02 - Estimated value: 172.08 | Target value: 172.08
Example #1.720789e+02 - Estimated value: 172.08 | Target value: 172.08
Example #1.720760e+02 - Estimated value: 172.08 | Target value: 172.07
Example #1.720731e+02 - Estimated value: 172.07 | Target value: 172.07
Example #1.720702e+02 - Estimated value: 172.07 | Target value: 172.07
Example #1.720674e+02 - Estimated value: 172.07 | Target value: 172.07
Example #1.720645e+02 - Estimated value: 172.06 | Target value: 172.06
Example #1.720616e+02 - Estimated value: 172.06 | Target value: 172.06
Example #1.720587e+02 - Estimated value: 172.06 | Target value: 172.06
Example #1.720558e+02 - Estimated value: 172.05 | Target value: 172.05
Example #1.720529e+02 - Estimated value: 172.05 | Target value: 172.05
Example #1.720500e+02 - Estimated value: 172.05 | Target value: 172.05
Example #1.720471e+02 - Estimated value: 172.05 | Target value: 172.05
Example #1.720442e+02 - Estimated value: 172.04 | Target value: 172.04
Example #1.720413e+02 - Estimated value: 172.04 | Target value: 172.04
Example #1.720385e+02 - Estimated value: 172.04 | Target value: 172.04
Example #1.720356e+02 - Estimated value: 172.03 | Target value: 172.03
Example #1.720327e+02 - Estimated value: 172.03 | Target value: 172.03
Example #1.720298e+02 - Estimated value: 172.03 | Target value: 172.03
Example #1.720269e+02 - Estimated value: 172.03 | Target value: 172.02
Example #1.720240e+02 - Estimated value: 172.02 | Target value: 172.02
Example #1.720211e+02 - Estimated value: 172.02 | Target value: 172.02
Example #1.720183e+02 - Estimated value: 172.02 | Target value: 172.02
Example #1.720154e+02 - Estimated value: 172.01 | Target value: 172.01
Example #1.720125e+02 - Estimated value: 172.01 | Target value: 172.01
Example #1.720096e+02 - Estimated value: 172.01 | Target value: 172.01
Example #1.720067e+02 - Estimated value: 172.01 | Target value: 172.00
Example #1.720038e+02 - Estimated value: 172.00 | Target value: 172.00
Example #1.720010e+02 - Estimated value: 172.00 | Target value: 172.00
Example #1.719981e+02 - Estimated value: 172.00 | Target value: 172.00
Example #1.719952e+02 - Estimated value: 171.99 | Target value: 171.99
Example #1.719923e+02 - Estimated value: 171.99 | Target value: 171.99
Example #1.719894e+02 - Estimated value: 171.99 | Target value: 171.99
Example #1.719865e+02 - Estimated value: 171.99 | Target value: 171.98
Example #1.719837e+02 - Estimated value: 171.98 | Target value: 171.98
Example #1.719808e+02 - Estimated value: 171.98 | Target value: 171.98
Example #1.719779e+02 - Estimated value: 171.98 | Target value: 171.98
Example #1.719750e+02 - Estimated value: 171.97 | Target value: 171.97
Example #1.719721e+02 - Estimated value: 171.97 | Target value: 171.97
Example #1.719693e+02 - Estimated value: 171.97 | Target value: 171.97
Example #1.719664e+02 - Estimated value: 171.97 | Target value: 171.96
Example #1.719635e+02 - Estimated value: 171.96 | Target value: 171.96
Example #1.719606e+02 - Estimated value: 171.96 | Target value: 171.96
Example #1.719578e+02 - Estimated value: 171.96 | Target value: 171.96
Example #1.719549e+02 - Estimated value: 171.95 | Target value: 171.95
Example #1.719520e+02 - Estimated value: 171.95 | Target value: 171.95
Example #1.719491e+02 - Estimated value: 171.95 | Target value: 171.95
Example #1.719462e+02 - Estimated value: 171.95 | Target value: 171.94
Example #1.719434e+02 - Estimated value: 171.94 | Target value: 171.94
Example #1.719405e+02 - Estimated value: 171.94 | Target value: 171.94
Example #1.719376e+02 - Estimated value: 171.94 | Target value: 171.94
Example #1.719347e+02 - Estimated value: 171.93 | Target value: 171.93
Example #1.719319e+02 - Estimated value: 171.93 | Target value: 171.93
Example #1.719290e+02 - Estimated value: 171.93 | Target value: 171.93
Example #1.719261e+02 - Estimated value: 171.93 | Target value: 171.92
Example #1.719233e+02 - Estimated value: 171.92 | Target value: 171.92
Example #1.719204e+02 - Estimated value: 171.92 | Target value: 171.92
Example #1.719175e+02 - Estimated value: 171.92 | Target value: 171.92
Example #1.719146e+02 - Estimated value: 171.91 | Target value: 171.91
Example #1.719118e+02 - Estimated value: 171.91 | Target value: 171.91
Example #1.719089e+02 - Estimated value: 171.91 | Target value: 171.91
Example #1.719060e+02 - Estimated value: 171.91 | Target value: 171.90
Example #1.719032e+02 - Estimated value: 171.90 | Target value: 171.90
Example #1.719003e+02 - Estimated value: 171.90 | Target value: 171.90
Example #1.718974e+02 - Estimated value: 171.90 | Target value: 171.90
Example #1.718946e+02 - Estimated value: 171.89 | Target value: 171.89
Example #1.718917e+02 - Estimated value: 171.89 | Target value: 171.89
Example #1.718888e+02 - Estimated value: 171.89 | Target value: 171.89
Example #1.718859e+02 - Estimated value: 171.88 | Target value: 171.88
Example #1.718831e+02 - Estimated value: 171.88 | Target value: 171.88
Example #1.718802e+02 - Estimated value: 171.88 | Target value: 171.88
Example #1.718773e+02 - Estimated value: 171.88 | Target value: 171.88
Example #1.718745e+02 - Estimated value: 171.87 | Target value: 171.87
Example #1.718716e+02 - Estimated value: 171.87 | Target value: 171.87
Example #1.718688e+02 - Estimated value: 171.87 | Target value: 171.87
Example #1.718659e+02 - Estimated value: 171.86 | Target value: 171.86
Example #1.718630e+02 - Estimated value: 171.86 | Target value: 171.86
Example #1.718602e+02 - Estimated value: 171.86 | Target value: 171.86
Example #1.718573e+02 - Estimated value: 171.86 | Target value: 171.86
Example #1.718544e+02 - Estimated value: 171.85 | Target value: 171.85
Example #1.718516e+02 - Estimated value: 171.85 | Target value: 171.85
Example #1.718487e+02 - Estimated value: 171.85 | Target value: 171.85
Example #1.718458e+02 - Estimated value: 171.84 | Target value: 171.84
Example #1.718430e+02 - Estimated value: 171.84 | Target value: 171.84
Example #1.718401e+02 - Estimated value: 171.84 | Target value: 171.84
Example #1.718373e+02 - Estimated value: 171.84 | Target value: 171.84
Example #1.718344e+02 - Estimated value: 171.83 | Target value: 171.83
Example #1.718315e+02 - Estimated value: 171.83 | Target value: 171.83
Example #1.718287e+02 - Estimated value: 171.83 | Target value: 171.83
Example #1.718258e+02 - Estimated value: 171.82 | Target value: 171.82
Example #1.718230e+02 - Estimated value: 171.82 | Target value: 171.82
Example #1.718201e+02 - Estimated value: 171.82 | Target value: 171.82
Example #1.718172e+02 - Estimated value: 171.82 | Target value: 171.82
Example #1.718144e+02 - Estimated value: 171.81 | Target value: 171.81
Example #1.718115e+02 - Estimated value: 171.81 | Target value: 171.81
Example #1.718087e+02 - Estimated value: 171.81 | Target value: 171.81
Example #1.718058e+02 - Estimated value: 171.80 | Target value: 171.80
Example #1.718030e+02 - Estimated value: 171.80 | Target value: 171.80
Example #1.718001e+02 - Estimated value: 171.80 | Target value: 171.80
Example #1.717972e+02 - Estimated value: 171.80 | Target value: 171.80
Example #1.717944e+02 - Estimated value: 171.79 | Target value: 171.79
Example #1.717915e+02 - Estimated value: 171.79 | Target value: 171.79
Example #1.717887e+02 - Estimated value: 171.79 | Target value: 171.79
Example #1.717858e+02 - Estimated value: 171.78 | Target value: 171.78
Example #1.717830e+02 - Estimated value: 171.78 | Target value: 171.78
Example #1.717801e+02 - Estimated value: 171.78 | Target value: 171.78
Example #1.717773e+02 - Estimated value: 171.78 | Target value: 171.78
Example #1.717744e+02 - Estimated value: 171.77 | Target value: 171.77
Example #1.717716e+02 - Estimated value: 171.77 | Target value: 171.77
Example #1.717687e+02 - Estimated value: 171.77 | Target value: 171.77
Example #1.717659e+02 - Estimated value: 171.76 | Target value: 171.76
Example #1.717630e+02 - Estimated value: 171.76 | Target value: 171.76
Example #1.717602e+02 - Estimated value: 171.76 | Target value: 171.76
Example #1.717573e+02 - Estimated value: 171.76 | Target value: 171.76
Example #1.717545e+02 - Estimated value: 171.75 | Target value: 171.75
Example #1.717516e+02 - Estimated value: 171.75 | Target value: 171.75
Example #1.717488e+02 - Estimated value: 171.75 | Target value: 171.75
Example #1.717459e+02 - Estimated value: 171.74 | Target value: 171.74
Example #1.717431e+02 - Estimated value: 171.74 | Target value: 171.74
Example #1.717402e+02 - Estimated value: 171.74 | Target value: 171.74
Example #1.717374e+02 - Estimated value: 171.74 | Target value: 171.74
Example #1.717345e+02 - Estimated value: 171.73 | Target value: 171.73
Example #1.717317e+02 - Estimated value: 171.73 | Target value: 171.73
Example #1.717288e+02 - Estimated value: 171.73 | Target value: 171.73
Example #1.717260e+02 - Estimated value: 171.73 | Target value: 171.72
Example #1.717231e+02 - Estimated value: 171.72 | Target value: 171.72
Example #1.717203e+02 - Estimated value: 171.72 | Target value: 171.72
Example #1.717174e+02 - Estimated value: 171.72 | Target value: 171.72
Example #1.717146e+02 - Estimated value: 171.71 | Target value: 171.71
Example #1.717118e+02 - Estimated value: 171.71 | Target value: 171.71
Example #1.717089e+02 - Estimated value: 171.71 | Target value: 171.71
Example #1.717061e+02 - Estimated value: 171.71 | Target value: 171.70
Example #1.717032e+02 - Estimated value: 171.70 | Target value: 171.70
Example #1.717004e+02 - Estimated value: 171.70 | Target value: 171.70
Example #1.716975e+02 - Estimated value: 171.70 | Target value: 171.70
Example #1.716947e+02 - Estimated value: 171.69 | Target value: 171.69
Example #1.716919e+02 - Estimated value: 171.69 | Target value: 171.69
Example #1.716890e+02 - Estimated value: 171.69 | Target value: 171.69
Example #1.716862e+02 - Estimated value: 171.69 | Target value: 171.68
Example #1.716833e+02 - Estimated value: 171.68 | Target value: 171.68
Example #1.716805e+02 - Estimated value: 171.68 | Target value: 171.68
Example #1.716777e+02 - Estimated value: 171.68 | Target value: 171.68
Example #1.716748e+02 - Estimated value: 171.67 | Target value: 171.67
Example #1.716720e+02 - Estimated value: 171.67 | Target value: 171.67
Example #1.716691e+02 - Estimated value: 171.67 | Target value: 171.67
Example #1.716663e+02 - Estimated value: 171.67 | Target value: 171.66
Example #1.716635e+02 - Estimated value: 171.66 | Target value: 171.66
Example #1.716606e+02 - Estimated value: 171.66 | Target value: 171.66
Example #1.716578e+02 - Estimated value: 171.66 | Target value: 171.66
Example #1.716549e+02 - Estimated value: 171.65 | Target value: 171.65
Example #1.716521e+02 - Estimated value: 171.65 | Target value: 171.65
Example #1.716493e+02 - Estimated value: 171.65 | Target value: 171.65
Example #1.716464e+02 - Estimated value: 171.65 | Target value: 171.64
Example #1.716436e+02 - Estimated value: 171.64 | Target value: 171.64
Example #1.716408e+02 - Estimated value: 171.64 | Target value: 171.64
Example #1.716379e+02 - Estimated value: 171.64 | Target value: 171.64
Example #1.716351e+02 - Estimated value: 171.63 | Target value: 171.63
Example #1.716323e+02 - Estimated value: 171.63 | Target value: 171.63
Example #1.716294e+02 - Estimated value: 171.63 | Target value: 171.63
Example #1.716266e+02 - Estimated value: 171.63 | Target value: 171.62
Example #1.716238e+02 - Estimated value: 171.62 | Target value: 171.62
Example #1.716209e+02 - Estimated value: 171.62 | Target value: 171.62
Example #1.716181e+02 - Estimated value: 171.62 | Target value: 171.62
Example #1.716153e+02 - Estimated value: 171.61 | Target value: 171.61
Example #1.716124e+02 - Estimated value: 171.61 | Target value: 171.61
Example #1.716096e+02 - Estimated value: 171.61 | Target value: 171.61
Example #1.716068e+02 - Estimated value: 171.61 | Target value: 171.60
Example #1.716040e+02 - Estimated value: 171.60 | Target value: 171.60
Example #1.716011e+02 - Estimated value: 171.60 | Target value: 171.60
Example #1.715983e+02 - Estimated value: 171.60 | Target value: 171.60
Example #1.715955e+02 - Estimated value: 171.59 | Target value: 171.59
Example #1.715926e+02 - Estimated value: 171.59 | Target value: 171.59
Example #1.715898e+02 - Estimated value: 171.59 | Target value: 171.59
Example #1.715870e+02 - Estimated value: 171.59 | Target value: 171.59
Example #1.715842e+02 - Estimated value: 171.58 | Target value: 171.58
Example #1.715813e+02 - Estimated value: 171.58 | Target value: 171.58
Example #1.715785e+02 - Estimated value: 171.58 | Target value: 171.58
Example #1.715757e+02 - Estimated value: 171.57 | Target value: 171.57
Example #1.715729e+02 - Estimated value: 171.57 | Target value: 171.57
Example #1.715700e+02 - Estimated value: 171.57 | Target value: 171.57
Example #1.715672e+02 - Estimated value: 171.57 | Target value: 171.57
Example #1.715644e+02 - Estimated value: 171.56 | Target value: 171.56
Example #1.715616e+02 - Estimated value: 171.56 | Target value: 171.56
Example #1.715587e+02 - Estimated value: 171.56 | Target value: 171.56
Example #1.715559e+02 - Estimated value: 171.55 | Target value: 171.55
Example #1.715531e+02 - Estimated value: 171.55 | Target value: 171.55
Example #1.715503e+02 - Estimated value: 171.55 | Target value: 171.55
Example #1.715474e+02 - Estimated value: 171.55 | Target value: 171.55
Example #1.715446e+02 - Estimated value: 171.54 | Target value: 171.54
Example #1.715418e+02 - Estimated value: 171.54 | Target value: 171.54
Example #1.715390e+02 - Estimated value: 171.54 | Target value: 171.54
Example #1.715362e+02 - Estimated value: 171.54 | Target value: 171.53
Example #1.715333e+02 - Estimated value: 171.53 | Target value: 171.53
Example #1.715305e+02 - Estimated value: 171.53 | Target value: 171.53
Example #1.715277e+02 - Estimated value: 171.53 | Target value: 171.53
Example #1.715249e+02 - Estimated value: 171.52 | Target value: 171.52
Example #1.715221e+02 - Estimated value: 171.52 | Target value: 171.52
Example #1.715192e+02 - Estimated value: 171.52 | Target value: 171.52
Example #1.715164e+02 - Estimated value: 171.52 | Target value: 171.51
Example #1.715136e+02 - Estimated value: 171.51 | Target value: 171.51
Example #1.715108e+02 - Estimated value: 171.51 | Target value: 171.51
Example #1.715080e+02 - Estimated value: 171.51 | Target value: 171.51
Example #1.715051e+02 - Estimated value: 171.50 | Target value: 171.50
Example #1.715023e+02 - Estimated value: 171.50 | Target value: 171.50
Example #1.714995e+02 - Estimated value: 171.50 | Target value: 171.50
Example #1.714967e+02 - Estimated value: 171.50 | Target value: 171.49
Example #1.714939e+02 - Estimated value: 171.49 | Target value: 171.49
Example #1.714911e+02 - Estimated value: 171.49 | Target value: 171.49
Example #1.714883e+02 - Estimated value: 171.49 | Target value: 171.49
Example #1.714854e+02 - Estimated value: 171.48 | Target value: 171.48
Example #1.714826e+02 - Estimated value: 171.48 | Target value: 171.48
Example #1.714798e+02 - Estimated value: 171.48 | Target value: 171.48
Example #1.714770e+02 - Estimated value: 171.48 | Target value: 171.48
Example #1.714742e+02 - Estimated value: 171.47 | Target value: 171.47
Example #1.714714e+02 - Estimated value: 171.47 | Target value: 171.47
Example #1.714686e+02 - Estimated value: 171.47 | Target value: 178.00
Gradient Descent in Practice
Let's now take a look at some techniques that make gradient descent (GD) work much better (more efficient). First, let's now investigate a technique called feature scaling, which will enable gradient descent to run faster.
- If we take look at the relationship between the range/magnitude/size of a feature (how big are the numbers in a feature) and range/magnitude/size of its associated parameter.
- Let's take the "prediction of house prices" problem as our example, where we will assume that we have only two features
The linear regression problem could be formulated as follows:
, where the parameter
represents the predicted price. The parameters
,
and b are the linear regression weights parameter, features vector and bias parameter, respectively. Note that we represent each feature corresponds to the
column of the training matrix
, where
and
. Spefically,
represents the size of a house (square feet, range: [300-2000])
is the number of bedrooms (range: [0 5]).
For this example, we can say that
takes realtively large range of values and
takes relatively small range of values. Now, let's take one training example: House:
,
, price = $500K for this training example what do you think the reasonable values for the size of the parameters
and
? 1) Let's look at one set of parameters:
,
,
. In this case, the estimated price becomes,
. It is clear that the estimated price is quite far from the actual price which was $500K. This is not a good set of parameter choices for
and
. 2) Let's take a look at another possibility
House:
,
,
, where the values of the weights are flipped. In this case, the estimated price becomes,
, which is a more reasonable estimate.- It is important to note that when a range of a feature is large (
in this case) it is more likely that a "good model" will choose a smaller value (such as
). Likewise, when the range of a feature is small then a "good model" will choose a larger weight value (such as
).
Feature Scaling in Practice
Question: How does this relate to gradient descent?
- Let's take a look at the scatter plot of the features,
and
and contour plot.
If we plot the training data, we can see that the
is in the much larger scale/range compared to
. - Next, let's look how the cost function might look in the contour plot.
We can see that in the contour plot
has much narrower range (say between 0 and 1) compared to
(say between 10 and 100). Thus, the contours form ovals/ellipses as they are in shorter one side and longer in the other. This is because very small change in
can have a very large impact on the estimated price (
), and that corresponds to a very large impact on the cost J as
tends to be multiplied with a very large number
(size in sq. feet). On the contrary, it takes significantly larger change in
to have a very large impact on the
and J as it tends to be multiplied with a small number
(# of bedrooms).
| size of the feature  | size of the parameter  |
size in sq. feet | Large | Small |
# of bedrooms | Small | Large |
The contour plot above is where what we will be dealing with if we decide to run the gradient descent algorithm on the example as is. As the cost function,
, w.r.t parameters,
and
, is tall and slim, gradient descent algorithm might be bouncing back and forth until it finds the solution, which makes the searching for the optimal point much longer and increases the possibility of divergence. In situations like this, useful thing to do is to scale the features, this means that some transformation in the input training data is employed. Consequently, the range for the parameters
and
will be limited to be in 0 and 1. The corresponding data points and cost function plot after the feature scaling becomes, Please note from the transformed data yields a contour plot that is more uniform and symmetrical (shapes more like circles). As a result, gradient descent can find a much more direct path to the global minimum compared to the tall and slim cost function in untransformed training data.
Recap: When you have different features with different range of values, it can cause the gradient descent algorithm to operate slowly. Rescaling the features into a comparable scale makes gradient descent run much faster.
Question: Why do we need feature scaling?
- It makes gradient descent algorithm operates faster
- It prevents the optimization from getting stuck in local optima
- It gives a better error/cost surface shape
- Machine learning algorithms just see numbers and doesn't know what the numbers represent — if there is a vast difference in the range say few ranging in thousands and few ranging in the tens, and the model might make the underlying assumption that higher ranging numbers have superiority of some sort. Thus, these more significant numbers start playing a more decisive role while training the model.
For instance,
Item | Weight (g) | Price ($) |
Orange | 15 | 1 |
Apple | 18 | 3 |
Banana | 12 | 2 |
Grapes | 10 | 5 |
Normally, weight cannot be compared with price as it won't yield any meaningful result, however, because of the large numbers in the weight column algorithm might assume that the weight is more important feature than the price. However, if we convert the weights to kg, then price becomes the more important feature.
To avoid that, we can employ feature scaling, where one significant number doesn’t impact the model just because of their large magnitude.
Question: But how do we actually rescale features into a comparable scale?
If we look at the data below, where the features
(room size in sq. feet) and
(number of bedrooms) is in the range of
and
, respectively. 1) Maximum value scaling: One way to scale is to take each
and
value is to divide them into the maximum value that each feature could reach, where
and
. Thus, the scaled
and
will be in the range of
and
, respectively. 2) Min-max scaling: Min-max scaling is one of the simplest methods to rescaling features into the interval
in its generalized form. To implement the min-max scaling we need the following operation. 3) Mean value scaling: Similar to the min-max scaling, you can also use the mean value for scaling the feature vector, which is called "mean normalization". Mean normalization rescales the datapoints in a way that they are symmetrically scattered around the origin,
where
and
. The mean value of the each feature could be calculated as
. If we assume that the mean value
is
, then the
will be in the range of
. Similarly, if the mean value of the feature
is
, the
will be in the range of
. 4) Z-score normalization: There is one last normalization method called z-score normalization, which utilizes mean and standard deviation, to standardize each feature (with the assumption of each feature is Gaussian distributed),
, where
,
and
and
. For the feature
with the mean and stadard deviation of
and
, the range of the
becomes
. Similarly, for the feature
with the mean and stadard deviation of
and
, the range of the
becomes
. As a rule of thumb in feature scaling, aim for about
for each feature
. These -1 and 1 values could be a little bit loose depending on the dataset and the application. For example,
,
,
⟹ Acceptable range
⟹ OK, no scaling needed
⟹ OK, no scaling needed
⟹ Too large, scaling needed
⟹ Too small, scaling needed
⟹ Too large, scaling needed
Question: With feature scaling, how are we going to know that the gradient descent algorithm finds the global minimum (converging)?
When running gradient descent, how can we be sure that it is converging (the algortihm is getting closer and closer to the global minimum of the cost function)? One of the key choices in gradient descent algorithm,
is the correct selection of the learning rate (α). Recall that the main objective of the gradient algorithm is:
. Therefore, if we plot the cost function (J) w.r.t the number of iterations, where each iteration means after the simultaneous updates of the parameters
and b. This figure is called the learning curve. If the gradient descent is working properly, the cost (J) should decrease after every iteration. If Jever increases from one iteration to other, that might mean that either α is chosen poorly (usually means that α is too large) or there might be a potential bug in the code. It is also important to note that the cost (J) is not decreasing much between the iterations 400 and 500, this means that the gradient descent is more or less converged.
Another way to decide whether your model is done training or not is the automatic convergence test,
- Let ϵ be

- If
decreases by
in one iteration, declare convergence (hopefully found parameters
and b to be close to the global minimum).
Let's now learn about how to choose an appropriate learning rate.
The gradient descent based learning algorithm will run much better with the selection of the appropriate learning rate. If it is too small, it might converge but run very slowly. However, if it is too large it might not even converge. It is important to note that with a small enough α, the cost function
should decrease on every iteration. If we set learning rate to be a very small value and still the cost function increases over number of iterations, it is a strong indicator of an implementation bug. - As a rule of thumb. values of learning rate to try:
, where each item is roughly 3 times larger than the previous one.
Let's have a practice to understand the concepts explanied above!
The goal of the practice is as follows:
- Explore the impact of learning rate on the gradient descent (GD) performance.
- Explore the impact of feature scaling on the GD performance.
Similar to the previous practices, we will use the housing price prediction dataset.
% Load House Prices Dataset
dataset = importdata('house_prices_dataset.txt');
X_train = dataset(:,1:4);
y_train = dataset(:,end);
disp(['Size X_train: [' num2str(size(X_train)) ']'])
disp(['Size y_train: [' num2str(size(y_train)) ']'])
X_train
1244 3 1 64
1947 3 2 17
1725 3 2 42
1959 3 2 15
1314 2 1 14
864 2 1 66
1836 3 1 17
1026 3 1 43
3194 4 2 87
788 2 1 80
Let's visualize the dataset and its features by plotting each feature against the price
plotDatasetAgainstFeatures(X_train,y_train,["size (sqft)","num bedrooms","num floors","age"],"Price ($1000s)")
Plotting the dataset features w.r.t. the targets gives us some hint of which features play a significant role on price. For instance, number of bedrooms and floors don't have a strong impact on price. However, we can infer that larger and newer houses have higher prices compared to smaller and older houses.
Let's display and compare the peak-to-peak range between the original and z-score normalized data
X_norm = (X_train-mu)./sigma;
tiledlayout(3,1,"TileSpacing","loose")
scatter(X_train(:,1),X_train(:,4),'MarkerFaceColor','flat')
title("Unnormalized Input")
scatter(X_mean(:,1),X_mean(:,4),'MarkerFaceColor','flat')
title("$X-\mu$","Interpreter","latex")
scatter(X_norm(:,1),X_norm(:,4),'MarkerFaceColor','flat')
title("$(X-\mu)/\sigma$","Interpreter","latex")
disp(['Peak-to-peak range of each column (original): [' num2str(max(X_train) - min(X_train)) ']'])
Peak-to-peak range of each column (original): [2406 4 1 95]
disp(['Peak-to-peak range of each column (normalized): [' num2str(max(X_norm) - min(X_norm)) ']'])
Peak-to-peak range of each column (normalized): [5.8157 6.1042 2.0459 3.6667]
Let's visualize the distribution of some features before and after the normalization
featureLabels = ["size (sqft)","num bedrooms","num floors","age"];
T = tiledlayout(1,length(featureLabels));
for i = 1:length(featureLabels)
h = histfit(X_train(:,i));
ylabel("Relative frequency")
title(T,"Feature distribution before the input normalization")
f.Position = [0 0 700 300];
T = tiledlayout(1,length(featureLabels));
for i = 1:length(featureLabels)
h = histfit(X_norm(:,i));
ylabel("Relative frequency")
title(T,"Feature distribution after the input normalization")
f.Position = [0 0 700 300];
It is important to note from above figures that the all the features are centered around 0 and they also share the same range which is roughly [-2 2]. Let's run our gradient descent algorithm with the normalized input and significantly larger lerning rate to see the input normalization effect.
alpha = 1.0e-1; % Learning rate
condStop = 1e-3; % Stopping condition
maxIter = 1000; % Maximum number of iterations
% Run BGD algoritm and print the results
[w_vec,b_vec,J_hist] = hBatchGradientDescentMV(X_norm, y_train, alpha, condStop, ...
maxIter, verbose, verboseFreq);
==================================
BGD Stopped: max number of iterations (999)
As can be seen from the results the normalized inputs yield a faster execution time
disp(['Elapsed time unnormalized input: ' num2str(tElapsed1)])
Elapsed time unnormalized input: 0.19128
disp(['Elapsed time normalized input: ' num2str(tElapsed2)])
Elapsed time normalized input: 0.050034
Let's plot the cost function
contour plot for the unnormalized and normalized features. tiledlayout(3,2,"TileSpacing","tight");
X = X_train(:,4); % Input data
% Plot the cost function without input normalization
w_vec = -1e4:50:1e4; % weights vector
b_vec = -1e4:50:1e4; % bias vector
% Plot the cost function before the input normalization
J = scanCostFunction(w_vec,b_vec,X,y);
contour(b_vec,w_vec,J,20);
%------------------------------------------------------------
% Max-value normalize the input feature
X_normalize = normalizeInput(X,"Max Value");
% Plot the cost function after input normalization
J_maxVal = scanCostFunction(w_vec,b_vec,X_normalize,y);
contour(b_vec,w_vec,J_maxVal,20);
%------------------------------------------------------------
% Mean-value normalize the input feature
X_normalize = normalizeInput(X,"Mean Value");
% Plot the cost function after input normalization
J_meanVal = scanCostFunction(w_vec,b_vec,X_normalize,y);
contour(b_vec,w_vec,J_meanVal,20);
%------------------------------------------------------------
% Min-max normalize the input feature
X_normalize = normalizeInput(X,"Min-max");
% Plot the cost function after input normalization
J_minMax = scanCostFunction(w_vec,b_vec,X_normalize,y);
contour(b_vec,w_vec,J_minMax,20);
%------------------------------------------------------------
% z-score normalize the input feature
X_normalize = normalizeInput(X,"z-score");
% Plot the cost function after input normalization
J_zScore = scanCostFunction(w_vec,b_vec,X_normalize,y);
contour(b_vec,w_vec,J_zScore,20);
f.Position = [0 0 750 750];
Let's plot two features
and
w.r.t the cost function (
) % Run BGD algoritm and print the results
[w_vec,b_vec,J_hist] = hBatchGradientDescentMV(X_norm,y_train,alpha,condStop,maxIter,verbose,verboseFreq);
==================================
BGD Stopped: max number of iterations (999)
T = tiledlayout(1,2,"TileSpacing","loose");
% Cost function without normalization
w1_vec = -1e4:10:1e4; % w1 vector
w2_vec = -1e4:10:1e4; % w2 vector
X_trainSub = [X_train(:,1) X_train(:,2)]; % two out of four features (subset)
J(i,j) = ( 1/(2*length(X_trainSub)) )*sum( ((w_subVec*X_trainSub.')+b_vec(end)-y_train.').^2 );
contour(w1_vec,w2_vec,J.',"ShowText",true,"LabelFormat","%0.1e m")
title("J(w,b) w/o norm.")
% Cost function with normalization
w1_vec = -1e4:10:1e4; % w1 vector
w2_vec = -1e4:10:1e4; % w2 vector
X_norm = [normalizeInput(X_train(:,1),"Mean Value") normalizeInput(X_train(:,2),"Mean Value")];
J(i,j) = ( 1/(2*length(X_norm)) )*sum( ((w_subVec*X_norm.')+b_vec(end)-y_train.').^2 );
contour(w1_vec,w2_vec,J.',"ShowText",true,"LabelFormat","%0.1e m")
title("J(w,b) w/ mean val. norm.")
f.Position = [0 0 700 400];
Learning Rate in Practice
We know that the learning rate (α) controls the size of the update in each iteration and it is shared by all the parameters. Let's run gradient descent and try a few settings.
Case 1: alpha = 9.9e-7
alpha = 9.9e-7; % Learning rate
condStop = 1e-3; % Stopping condition
maxIter = 1e4; % Maximum number of iterations
% Run BGD algoritm and print the results
[w_hist,b_hist,J_hist] = hBatchGradientDescentMV(X_train,y_train,alpha,condStop,maxIter,verbose,verboseFreq);
==================================
BGD Stopped: max number of iterations (9999)
w_vec = -3:0.001:3; % weights vector
J = scanCostFunction(w_vec,b_vec,X_train(:,1),y_train);
tiledlayout(1,2,"TileSpacing","loose")
plot(1:length(J_hist),J_hist,'b','LineWidth',3);
ylabel("Cost function, J(w,b)");
title("Cost vs Iteration")
plot(w_hist(1,:),J_hist,'mo-','LineWidth',2)
plot(w_vec,J,'b','LineWidth',3)
ylabel("Cost function, J(w,b)");
It appears the learning rate is too high. The solution does not converge. Cost is increasing rather than decreasing.
Case 2: alpha = 9e-7
alpha = 9e-7; % Learning rate
condStop = 1e-3; % Stopping condition
maxIter = 1e4; % Maximum number of iterations
% Run BGD algoritm and print the results
[w_hist,b_hist,J_hist] = hBatchGradientDescentMV(X_train,y_train,alpha,condStop,maxIter,verbose,verboseFreq);
==================================
BGD Stopped: max number of iterations (9999)
w_vec = -3:0.001:3; % weights vector
J = scanCostFunction(w_vec,b_vec,X_train(:,1),y_train);
tiledlayout(1,2,"TileSpacing","loose")
plot(1:length(J_hist),J_hist,'b','LineWidth',3);
ylabel("Cost function, J(w,b)");
title("Cost vs Iteration")
plot(w_hist(1,:),J_hist,'mo-','LineWidth',2)
plot(w_vec,J,'b','LineWidth',3)
ylabel("Cost function, J(w,b)");
Cost is decreasing throughout the run showing that alpha is not too large.
Case 2: alpha = 1e-7
alpha = 1e-7; % Learning rate
condStop = 1e-3; % Stopping condition
maxIter = 1e4; % Maximum number of iterations
% Run BGD algoritm and print the results
[w_hist,b_hist,J_hist] = hBatchGradientDescentMV(X_train,y_train,alpha,condStop,maxIter,verbose,verboseFreq);
==================================
BGD Stopped: max number of iterations (9999)
w_vec = -3:0.001:3; % weights vector
J = scanCostFunction(w_vec,b_vec,X_train(:,1),y_train);
tiledlayout(1,2,"TileSpacing","loose")
plot(1:length(J_hist),J_hist,'b','LineWidth',3);
ylabel("Cost function, J(w,b)");
title("Cost vs Iteration")
plot(w_hist(1,:),J_hist,'mo-','LineWidth',3)
plot(w_vec,J,'b','LineWidth',3)
ylabel("Cost function, J(w,b)");
ylim([min(J) max(J)/1000])
You can see from the figures that
is decreasing without crossing the minimum. This solution will also converge, though not quite as quickly as the previous example. Local Function Definitions
function plotDatasetAgainstFeatures(X_train,y_train,Xstring,ystring)
if size(X_train,2) ~= numel(Xstring)
error("Training data and respective string sizes are not matching.")
if size(y_train,2) ~= numel(ystring)
error("Targets and respective strings vector sizes are not matching.")
% Create the feature plots
T = tiledlayout(1,size(X_train,2),"TileSpacing","loose");
for i = 1:size(X_train,2)
plot(X_train(:,i),y_train,'o','MarkerFaceColor','b')
f.Position = [0 0 900 300];
function normInput = normalizeInput(input,normType)
if strcmpi(normType,"Max Value")
normInput = input./max(input);
elseif strcmpi(normType,"Min-max")
normInput = ( (input - min(input))/(max(input)-min(input)) )*(b-a) + a;
elseif strcmpi(normType,"Mean Value")
meanInput = (1/length(input))*sum(input);
normInput = ( (input - meanInput)/(max(input)-min(input)) );
elseif strcmpi(normType,"z-score")
meanInput = (1/length(input))*sum(input);
varInput = (1/(length(input)-1))*sum((input-meanInput).^2); % sample variance (N-1)
normInput = (input - meanInput)/sqrt(varInput);
error("Undefined input normalization type.")
function J = scanCostFunction(w_vec,b_vec,X,y)
% Calculate the cost function values
J(i,j) = (1/(2*length(X))).*sum((w.*X+b-y).^2);